Data scientists build software that transforms data and produces models or predictions, so they need basic practices such as version control and clear naming.
2
A project template can split a data flow into small scripts with explicit inputs and outputs, allowing Python, R, or other languages to work together without hiding the process.
3
The right amount of software structure depends on the final use of the project, while reproducibility, data checks, and repository security should be handled throughout the workflow.
Summary
Rodolfo Núñez explains why data scientists need basic software practices when their work must become a useful product. He argues for Git, clear names, reproducible workflows, and a documented project template that new hires learn through a standard assignment. His team structures projects as data flows made from small scripts. Each script has a defined input and output, a numbered prefix, and one main task. Pipelines then call the scripts in order, which makes failures easier to locate and lets an engineer run the project with a single command. Since the scripts are independent, the team can mix Python, R, and other languages. Núñez also compares this approach with Jupyter Notebook, discusses data checks with Great Expectations, criticizes the elbow method for choosing clusters, and describes cleaning sensitive material from Git repositories. The conversation stays practical. He accepts simpler code when a project only needs to produce an output, and asks for more structure when others will reuse the result.
Data scientists need software basics because their work becomes a product
Rodolfo Núñez says data scientists build software that takes data, transforms it, and produces a model, prediction, or another useful output. He does not expect them to become professional software engineers, but he does expect basic practices. Git, or another version-control system such as SVN or Mercurial, is his first requirement. He rejects files named things such as "Untitled one" and "Untitled two" because they do not support reproducibility. He also asks people to name variables and functions clearly. For booleans, he uses prefixes such as "is" and "has" so readers can tell that the value answers a yes-or-no question.
Clear names reduce the need to explain code elsewhere
Núñez prefers code that documents itself through names rather than relying heavily on comments. He gives data frames descriptive names that record their role, such as a frame containing cleaned missing values, instead of naming every frame "df". This matters when several objects live in a long script and are used at different frequencies. A name can tell another data scientist or machine learning engineer what an object contains and what happened to it. He still discusses comments as a valid school of thought, but his practice is to make variables, functions, and other objects understandable when someone reads the code.
Onboarding can teach junior data scientists how a team works
Núñez describes an onboarding process for junior data scientists who may come from school or a boot camp with experience in notebooks but little experience with Git. The process is documented and ends with a standard assignment. New hires create their own repository and push the work, which gives them practice with basic Git operations without requiring advanced knowledge of pull requests. After that, they receive the team's in-house data science project template. Núñez says every team should have a template suited to its own work. The template gives new members a shared way to organize projects and makes code review more consistent.
The final use of a project determines how much engineering structure it needs
When Abi Aryan asks where to use unit tests, integration tests, classes, and abstractions, Núñez answers that the final objective should decide. A team can spend so much time making a project perfect that it never finishes. Many projects on his team are a series of scripts that produce an output for production, and those projects do not always need classes or object-oriented programming. A reusable tool for other data scientists needs more general structure because many people will use it. A script that calculates a score and sends it onward may need much less. The standard is whether the result can be understood, transferred, and put into use.
R Markdown and Quarto make mixed-language work easier to manage
Núñez contrasts Jupyter Notebook with R Markdown and Quarto, drawing on his experience with RStudio and Python. A Jupyter Notebook stores cell outputs inside the notebook file, which can make the file a large JSON document containing plots and data. Committing changed outputs can consume repository space, and an open repository can accidentally receive private data displayed in a cell. In R Markdown and Quarto, the source file contains the written document and code while outputs are handled separately. These formats also allow code cells for Python, R, C++, and other languages in one document, rather than requiring one notebook kernel for the whole file.
A numbered script pipeline makes a data flow traceable
Núñez's project template treats data broadly. It includes data frames, CSV files, images, configuration files, and other stored inputs. Scripts pass outputs to later scripts, and each script has one main job. Preprocessing might read one dataset, read another, merge them, and clean missing values in separate steps. Output files receive prefixes such as "a00", "a01", and "a02", which identify the script that produced them. If a pipeline fails, the team knows which step to debug and which earlier outputs are available. A pipeline file calls the scripts in series or parallel, records timing information, and gives the README a simple command for running the project.
Independent scripts allow different languages to work in one project
Because each script has inputs and outputs, Núñez says a Python script can pass its result to an R script, which can pass a further result to another language. The language choice belongs to the specific part of the problem. He puts the project in a container and uses separate virtual environments for Python and R. A machine learning engineer does not need to understand every internal step if the data scientist has structured the project and documented how to run it. In production, the engineer can run Docker with the selected pipeline. This separates the details of implementation from the command needed to execute the workflow.
Data checks and repository cleanup address different failure risks
Núñez says his team is beginning to use Great Expectations to detect changes in data quality and structure. Planned checkpoints include the raw data at the start of a model pipeline, the data given to the model after preprocessing, and the model's output. The checks can cover columns, data types, and output distributions. He also describes security work across Bitbucket repositories. A password or sensitive dataset can remain in Git history even after a file is removed. His team used BFG to remove oversized files or specific commits and wrote a Python script to search repositories for terms such as "pass" and "PWD". They then used R Markdown for reports and force-pushed cleaned repositories where necessary.
The elbow method can change its answer when the plot resolution changes
Núñez calls the elbow method an illusion for choosing the number of clusters. His example uses a curve created from an exponential function. If someone tests one through five groups, the plotted points can suggest one optimum. Testing twenty groups changes the visual perspective and can move that apparent optimum. Testing one hundred groups can change it again. He recommends selecting plausible cluster counts and measuring the resulting models with the metric that matters for the task, such as error, precision, or recall. The choice should come from the model's measured performance rather than from where a plotted curve appears to bend.
