Podcast

Data Engineering in the Federal Sector

Shane Morris, DevisEpisode 223 · 1:03:23 · Apr 2024 · 532 viewsHosted by Demetrios Brinkmann
Thumbnail for Data Engineering in the Federal Sector Watch on YouTube
TL;DR
  1. 1

    Shane Morris argues that data engineering will remain in demand because AI systems need clean, well-managed data.

  2. 2

    Federal technology projects move slowly because tools, software versions, and architecture changes must pass security and authorization processes.

  3. 3

    His free data engineering curriculum aims to help beginners become employable through Git, Docker, SQL, Python, ETL, dashboards, and a capstone project.

Summary

Shane Morris discusses his path from music management into data engineering, federal contracting, and advising technology companies. His music work involved analytics, digital assets, cloud services, and early data pipelines, which helped him identify artists and regional audiences before they became widely known. He explains that federal projects differ from commercial work because tools and software versions must pass security reviews, FedRAMP requirements, and stakeholder approval. Engineers also have to explain technical changes to leaders whose expertise may be in biology, earth science, or military operations. Morris describes DataX, an autonomous decision product designed for devices with many sensors, and explains why older academic research can become useful when infrastructure and market needs catch up. He also describes his donation-based data engineering course, which covers practical fundamentals and ends with an interview-style capstone. He expects AI systems to create sustained demand for people who can build and maintain data pipelines.

Key ideas
03:10

Morris got his new job through adjacent communities and visible work

Shane Morris says he found his new role by going to places where he did not expect to belong. He met someone through TikTok, then ended up in a fantasy football league with data science and machine learning professionals. After he became available for work, they spoke and he received an offer within about 48 hours. Morris connects this to the value of showing up in adjacent markets rather than waiting for a perfectly matched opportunity. He also uses LinkedIn and TikTok to share his experiences, which gave Demetrios Brinkmann a way to discover his unusual mix of music, software, and data work.

06:05

Music management gave Morris practical experience with digital data

Morris explains that music management involved much more data work than people might expect. Music was already a digital asset, with streams, user accounts, analytics, e-commerce, and ticketing. At Earmilk, the team hosted music before SoundCloud existed and built its own analytics dashboard. Those analytics helped Morris identify where artists had local audiences and recommend support acts to talent buyers. He also describes using cloud virtual machines and many accounts to manipulate SoundCloud plays, an example of the rule-free environment he encountered at the time. The work taught him to combine software skills with audience and market knowledge.

13:57

Music has far less consumer spending than adjacent technology markets

Morris compares the money available in music with spending on technology hardware. He says Spotify made $17 billion in the prior year, while Apple made $17 billion selling AirPods alone. Music revenue is divided across streaming, tickets, merchandise, publishing, and other sources, and most fans spend relatively little on it. Morris left music after a six-month project for a shoe company paid what his annual salary at Sony Music Nashville had been. He says merchandise bought at concerts gives artists a better margin because fulfillment costs are lower than for online sales. He still recommends buying it directly at shows.

17:57

Federal technology changes require authorization and security review

Morris describes his time at Booz Allen Hamilton, where changing an agency's architecture required much more than showing a commercial benefit. Depending on the agency, the work could involve an authority to operate, FedRAMP status, and security classification requirements. FedRAMP levels determine where software can be used, and even agencies such as NASA can operate classified environments. If a desired tool is not approved, teams must find an alternative. Software upgrades also require review. Morris says that moving from one Python version to another can require testing and a new security process because stakeholders need confidence that the change will not create unexpected effects.

20:40

Federal engineers often explain technical systems to non-technical decision-makers

Morris says the people approving federal systems may have deep expertise in areas such as earth science, biology, or military operations rather than computer science. The federal government also has a talent gap because software professionals can often earn more in the private sector. Engineers therefore need to explain tools in terms that match the audience. Morris describes getting Databricks approved for a Navy human resources project after it became FedRAMP-approved. He had to explain what the tool would help the organization do, rather than rely on terms such as application-level dashboards or a single source of truth.

27:03

DataX applies autonomous decision-making to sensor-rich devices

Morris describes DataX from Decision Zone as a distributed autonomous decision product that can run on devices and chipsets. It is based on a programming language called Rapid, which began at Stanford in 1996 and was later compiled in Java. The language was designed for parallel processing and places business logic in an unusual structure. Morris gives small defense drones as one possible use case. Instead of requiring a human pilot for every decision, autonomous agents could self-pilot, correct their behavior, and complete missions in difficult conditions. He also sees uses wherever many sensors need fast, local decisions.

37:05

Federal approval depends on mission outcomes rather than commercial return

Morris says commercial leaders often evaluate a tool through profit, cost savings, or time saved. Federal and military leaders frame the same decision around mission completion. For a Navy human resources system, the relevant outcomes include recruiting officers, retaining them, filling difficult billets, and helping military families relocate. This changes how an engineer presents a technology choice. Morris says Databricks gained support when leaders understood its connection to the Navy's modernization needs, even though they did not work through the technical details. He credits leaders who trust specialists and accept commercial tools when they fit the mission.

45:16

Data governance becomes easier to explain through a hospital record

Morris explains data governance with a hospital example. A person's information should follow them between departments, with allergies, doctors, dates, tags, and changes recorded correctly. Data lineage shows how the record moved and who changed it. He applies the same idea to an F-35 supplier chain: if multiple aircraft have a defect, lineage can identify a shared supplier and production date. Governance also includes permissions, since doctors and nurses may have different rights to change information. Morris says the exact implementation differs by organization, but consistency, history, ownership, and authorized changes are common needs.

50:35

His free course focuses on employable data engineering fundamentals

Morris created a donation-based curriculum because he did not want cost to block people from entering data engineering. His target is a junior engineer who can build an ETL pipeline and a basic dashboard. The course covers Git, Docker, SQL, and Python, then points learners to additional material. He also notes that some public libraries provide access to premium Udemy subscriptions. His plan is to prepare learners for an interview-style review after roughly six months, where they explain a capstone project built around a subject they know well. Morris believes that project can demonstrate practical ability in a corporate setting.

"You're going to have to have immaculate data to put in these models and that means that you're going to need a lot of quality data engineers."Shane Morris54:22
Who should watch
  • You are moving from software or analytics into federal contracting and want to understand why ordinary architecture and upgrade decisions take longer.
  • You need to explain a data platform to agency or military stakeholders who care about mission outcomes rather than commercial return.
  • You are starting data engineering and want a practical path built around fundamentals, a portfolio project, and interview preparation.