Python4DataScience

python-basics-tutorial.readthedocs.ioLegacy softwareThere may be various reasons for wanting to modify existing software: to add a feature, to fix a bug, to improve the software architecture (→ Refactoring), to improve resource usage. If an existing...

In this week’s workshop, we applied TDD to legacy software. We also wrote down what lessons we can take away for other projects: python-basics-tutorial.readthedocs.io/en/latest/te... #TDD #Legacy #Software #DX

python4data.scienceGit branches$ git branch [-a] [-l " GLOB_PATTERN"], shows all local branches in a repository.--a, also shows all removed branches.,,-l, restricts the branches to those that correspond to a specific p...

Git 2.56 introduced a number of exciting features that improve the way we work with merges: 'git add --resolved' and 'git merge-base'. www.python4data.science/en/latest/pr... www.python4data.science/en/latest/pr... #Git #DeveloperExperience #DX

python4data.scienceCite dataSimilar to the way in which researchers routinely provide bibliographical references to other academic sources, data should also be cited. Although data is frequently shared, it is not always cited...

Having first taken a closer look at CSVW for table dialects and schemas, we have now examined data resource and data package formats in detail, so that we can publish and cite data more effectively: www.python4data.science/en/latest/pr... #Python #DataScience

python4data.scienceCSVOverview:,,, Data structure support,--, CSV is used to store tabular data, but unlike other serialisation formats reviewed here, it’s not suitable for (nested) objects.,, Standardisation,--, CSV is...

21 Sept

CSVW is a metadata standard for CSV files. If it becomes widely adopted, it will make processing CSV files so much better: www.python4data.science/en/latest/da... #Python #DataScience

python4data.scienceCleaning and validating data‘Garbage in, garbage out’ is a stark reminder that it is almost impossible to draw meaningful insights from poor-quality data. Whilst there are highly regulated, safety-critical sectors in which da...

18 Sept

We have significantly revised the section on data validation and cleansing and expanded it to include data quality criteria and processes: www.python4data.science/en/latest/cl... #Python #DataScience

python4data.scienceString comparisonsdifflib: You can compare strings using the get_close_matches option of Python’s built-in difflib library. If you want to specify the minimum match score required for a suggestion to be displayed, y...

3 Sept

We have updated the section on text comparison – featuring difflib, TheFuzz and textacy: www.python4data.science/en/latest/cl... #Python

python-basics-tutorial.readthedocs.ioDecoratorsFunctions can also be passed as arguments to other functions, and the results of other functions can be returned. For example, it is possible to write a Python function that takes another function ...

18 Aug

At first, it can sometimes be difficult to imagine a practical use for decorators. We will therefore now present a range of different examples: python-basics-tutorial.readthedocs.io/en/latest/fu... #Python #Decorators

python4data.scienceParallelise pandasIn Enhancing performance, some possibilities are described for improving the performance of pandas. However, there are also special libraries that can parallelise the processing of data frames. cuD...

11 Aug

Narwhals provides a lightweight and extensible compatibility layer between various DataFrame libraries: www.python4data.science/en/latest/pe... #Python #pandas #DataFrame

python-basics-tutorial.readthedocs.ioMockIn this chapter, we will test the CLI. To do this, we will use the mock library, which has been included as part of the standard Python library since Python 3.3 under the name unittest.mock. Object...

5 Aug

We have expanded the section on mocks for testing and added spies: python-basics-tutorial.readthedocs.io/en/latest/te... #Python #Testing

python4data.scienceSecurityAs of July 2026, PyPI hosts over 750,000 packages, and this number is growing daily. An average Python project includes dozens of additional dependencies – packages that you have never explicitly c...

3 Aug

🧵 1/2 Attacks on the software supply chain are increasing rapidly. We have therefore updated our guide on how to make your projects more secure, adding answers to the following questions:

3 Aug

• How can you make your code more secure? • How can you ensure separation of concerns? • How can you continuously verify the security of your dependencies? • How do you create SBOMs? www.python4data.science/en/latest/pr... #Python #SupplyChain #Vulnerability #SBOM

python-basics-tutorial.readthedocs.ioshot-scrapershot-scraper is a tool to automate the process of updating screenshots. Installation: Use: shot-scraper can be used in several ways …for single screenshots on the command line: …or with additional ...

24 July

Did you know that with shot-scraper video you can explain your user interface using a storyboard: python-basics-tutorial.readthedocs.io/en/latest/do... #Python #Docs #Video

python4data.scienceSoftware Bill of Materials (SBOM)A Software Bill of Materials (SBOM) is a document used to exchange information about software and its composition. This format is primarily used in the security field to check software and its depe...

22 July

Software Bill of Materials (SBOMs) are commonly used to check software and its dependencies for vulnerabilities using CVE and OSV. We have described how to create these for Python packages here: python4data.science/en/latest/pr... #Python #SBOM #Vulnerability

agentic-software-engineering.readthedocs.ioSpec-Driven DevelopmentDevelopment teams are increasingly facing challenges regarding predictability and maintainability when requirements and context are stored exclusively in short-lived chat threads. To address this i...

20 July

Specification-Driven Development (SDD) was developed to improve the predictability and maintainability of agentic software development. In particular, we are continuing to give OpenSpec a chance: agentic-software-engineering.readthedocs.io/en/latest/pr... #CodingAgent #SDD #DX

python4data.scienceprekprek is a variant of pre-commit, written in Rust, for managing and maintaining multilingual commit hooks. An essential task is to make the same scripts available to the entire development team. pre...

15 July

We are switching from pre-commit to prek. Whilst prek is also faster, the decisive factors were the security-related features: pinned SHA revisions and cooldown: www.python4data.science/en/latest/pr... #Python #DX #DevOps #DevSecOps #Security

python-basics-tutorial.readthedocs.ioFiles and Directoriespathlib implements path operations using pathlib.PurePath and pathlib.Path objects. The os and os.path modules, on the other hand, provide functions that work at a low level with str and bytes, whi...

13 July

We have added the `errors` argument to the instructions for reading and writing files, so that files with encoding issues can be read more effectively. And, fortunately, UTF-8 becomes the default in Python 3.15: python-basics-tutorial.readthedocs.io/en/latest/sa... #Python

python4data.sciencePython for Data ScienceThis is a tutorial on Data Science with Python. This immediately raises the question: What is Data Science? The term has become ubiquitous, but there is no single definition. Some even consider the...

8 July

We have released a new version. As well as the usual software updates, it includes agent-based code generation and securing the software supply chain: www.python4data.science/en/26.1.0/ #Python #DataScience #AgenticCoding #Claude #SoftwareEngineering #SoftwareSupplyChain #SupplyChain

Python4DataScience reposted

python4data.scienceSecurityIn previous chapters, we have already provided some tips designed to help ensure safer operation. Here, we would like to summarise and expand on the individual elements once again. We will be using...

25 June

#TIL that there is also a pre-commit check for uv audit: www.python4data.science/en/latest/pr... #Python #CVE #Vulnerability #Git #ITSecurity #SupplyChain

Python4DataScience reposted

agentic-software-engineering.readthedocs.ioGuardrailsThe term AI guardrails first appeared in 2023, when ChatGPT launched its consumer service. The initial versions contained numerous security and privacy vulnerabilities, which ultimately prompted ma...

23 June

Guardrails enable you to defend against prompt injection or jailbreak attacks by your coding agents. In our tutorial, we show you the options available to you for this purpose: agentic-software-engineering.readthedocs.io/en/latest/se... #PromptInjection #Jailbreak #AI #Agentic #Coding #ITSecurity

Python4DataScience reposted

agentic-software-engineering.readthedocs.ioSkillsAs coding agents evolve from simple chat interfaces towards autonomous task execution, context engineering has become a critical challenge. Agent Skills provide an open standard for modularising co...

22 June

We have added a section on @tiangolo.com’s Library Skills to our agent-based software engineering tutorial: agentic-software-engineering.readthedocs.io/en/latest/sh... #AI #Agentic #Coding #Skills

11 June

We have expanded our tutorials to include answers to the question of how coding agents can assist data scientists with programming: • python-basics-tutorial.readthedocs.io/de/latest/li... • github.com/veit/python-... #Python #DataScience #CodingAgents #Claude #Cursor

10 June

With uv≥0.11.19, you can check dependencies in your project for known vulnerabilities: • python-basics-tutorial.readthedocs.io/en/latest/li... • www.python4data.science/en/latest/pr... #Python #uv #Vulnerability #ITSecurity

Python4DataScience reposted

python-basics-tutorial.readthedocs.ioAdding more Python librariesAlthough Python’ „Batteries included“ philosophy means that you can already do a lot with the default installation of Python, there will inevitably come a situation where you need functionality tha...

1 June

If uv is not available in a Python environment, you can still specify the versions using pylock.toml. We have described how to do this here: python-basics-tutorial.readthedocs.io/en/latest/li... #Python #DevSecOps #ITSecurity

22 May

Thanks to Ruff, our Python4DataScience tutorial is now much more consistent: www.python4data.science/en/latest/ #Python

Python4DataScience reposted

linux-magazine.comKeep It Free » Linux MagazineLinux users associate open source licenses with software, but a bevy of licenses are available for documents, images, audio/video, fonts, and hardw...

7 May

Yesterday, the article by Frank Hofmann and me on open licences for documents, images, audio, video, fonts and hardware was published in Linux Magazine: www.linux-magazine.com/Issues/2026/... #OpenSource #OpenHardware

python4data.scienceRewriting historyThere are several commands in Git for rewriting history. git rebase -i is the best known and most flexible: you can reorder, merge, edit and remove commits. However, this flexibility comes with a d...

24 Apr

We have integrated two features from Git 2.54: • Git history: www.python4data.science/en/latest/pr... • Configuration-based Git hooks: www.python4data.science/en/latest/pr... #Git

python-basics-tutorial.readthedocs.ioTestingBasically, a distinction is made between static and dynamic test procedures. Static test procedures, are used to check the source code, although this is not executed. They are divided into- reviews...

23 Apr

The testing section has been extensively revised: • Test examples for cusy.tasks • Added Hypothesis extensions • Agile software development using test-driven development and behaviour-driven development python-basics-tutorial.readthedocs.io/en/latest/te... #Python #pytest #Agile #TDD #BDD

16 Apr

Today is our last day at PyConDE (@pyconde.bsky.social). We’re looking forward to having lots more interesting conversations with you, whether there are any new developments in research software using Python or on LLM coding agents #PyConDE #RSE #Python #AI #LLM

python-basics-tutorial.readthedocs.ioPublish packageFinally, you can deploy the package on the Python Package Index( PyPI) or another index, for example GitLab Package Registry or devpi. For the Python Package Index, you must register with Test PyPI...

11 Apr

Final preparations for PyConDE & PyData next week in Darmstadt. We’d love to see you there. Please feel free to arrange a meeting with us: cusy.io/en/about/eve... @pyconde.bsky.social #Python #PyData

We have analysed the attacks on the software #SupplyChain of LiteLLM and Telnyx and now recommend dependency cooldowns alongside immutable references, trusted publishers and digital attestations to secure deployments: • python-basics-tutorial.readthedocs.io/en/latest/pa... #Python #ITSec #DevSecOps

Python4DataScience reposted

research-and-innovation.ec.europa.euA new era for Open Research EuropeEuropean Commission to co-fund new phase of operation for open access publishing platform.

26 Mar

In future, Open Research Europe (ORE) will no longer be open solely to projects funded under the EU Programmes for Research; instead, all research institutions in the participating countries will be able to use the platform free of charge: research-and-innovation.ec.europa.eu/news/all-res...

python4data.scienceOpen dataA topic-based overview of public repositories containing research data. Agricultural sciences: AQUASTAT Dissemination System, Global information system of the Food and Agriculture Organization of t...

23 Mar

We have now significantly expanded our collection of open data and organised it by topic: www.python4data.science/en/latest/da... #Python #DataScience #OpenData #Agriculture #Biology #Chemistry #Climate #Weather #ComputerNetwork #Energy #Finance #GeoScience #Healthcare #ImageProcessing #Medicine

Python4DataScience reposted

dfg.deArtificial Intelligence in the Review Process

12 Mar

The German Research Association (@dfg.de) has established a binding framework for the use of AI. The aim is to ensure the legally compliant and transparent use of AI: www.dfg.de/en/news/news... #Research #RSE #Science #DataScience #ArtificialIntelligence #AI #Compliance

11 Mar

We are a community sponsor and would love to meet you at the conference. #Python #DataScience

PyCon DE & PyData@pyconde.bsky.social

📅 The PyCon DE & PyData 2026 schedule is live! Explore the talks and start planning your conference days in Darmstadt. pretalx.com/pyconde-pyda...

python-basics-tutorial.readthedocs.iotoxtox is an automation tool that works similarly to a CI tool, but can be run both locally and in conjunction with other CI tools on a server. In the following, we will set up tox for our Items appli...

26 Feb

Since tox 4.44.0, tox.ini has been frozen. Only the toml configuration is supported for further development. We have therefore updated our tutorial accordingly: python-basics-tutorial.readthedocs.io/en/latest/te... #Python #testing #tox

python-basics-tutorial.readthedocs.ioConfigurationYou can use configuration files to change the way pytest runs. If you repeatedly use certain options in your tests, such as--verbose or--strict-markers, you can store them in a configuration file s...

3 Mar

… and now we have also moved the pytest configuration to the pyproject.toml file: python-basics-tutorial.readthedocs.io/en/latest/te... #Python #Testing #pytest

cusy.ioHow LLM agents endanger open-source projectsVeit Schiele 17 February 2026 11–14 minutes At the beginning of the year, the threat posed to an open source project by LLM agents became known in a discussion on GitHub about Tailwind CSS. Open so...

18 Feb

We are pleased that we were able to develop an idea on how to better deal with LLM agents in the future.

Veit Schiele@veit-schiele.de

I have been thinking about how LLM agents pose a threat to open source projects and what strategies can offer us at least some protection. Nevertheless, this is likely to remain a challenge: cusy.io/en/blog/how-... #OpenSource #Ai #LLM #AISlop

python4data.scienceMemrayMemory usage is difficult to control in Python projects because the language does not explicitly indicate where memory is allocated, module imports can significantly increase consumption, and it is...

11 Feb

We analyse the memory consumption of our applications with memray and also monitor it continuously with pytest-memray: python4data.science/en/latest/pe... #Python

python4data.scienceGit InternalsSo far, we have looked at how you can use Git to manage the different states of your code. Now we want to show you the data and storage models that underlie Git. Data Model: You will be able to use...

9 Feb

Git 2.53 provides faster insights into the repository structure with 'git repo structure'. However, to better understand this, it is helpful to be more familiar with the Git data and storage models: www.python4data.science/en/latest/pr... #Git

python4data.sciencePerformancePython can be used to write and test code quickly because it is an interpreted language that types dynamically. However, these are also the reasons it is slow when performing simple tasks repeatedl...

19 Jan

The section on performance measurements and finding bottlenecks has been significantly expanded to include cProfile/profiling.tracing, tprof, and profiling.sampling/Tachyon: www.python4data.science/en/latest/pe... #Python #Performance

python-basics-tutorial.readthedocs.ioDocumentIn order for your software package to be useful, documentation is required that describes how your software can be installed, operated, used and improved: Those who want to use your package need in...

16 Jan

We have updated the documentation section with references to README, CONTRIBUTING, CHANGELOG, etc. python-basics-tutorial.readthedocs.io/en/latest/do... #Python #Documentation

python4data.scienceExtensionsAdministration: SQLAlchemy Admin for Starlette/FastAPI, Flexible admin interface for SQLAlchemy models. Downloads Contributors Commit activity Licence,, Piccolo Admin, Simple but powerful admin int...

7 Jan

We have updated the FastAPI extensions. It is very surprising to us that millions of extensions are being downloaded that have not been updated for over a year. www.python4data.science/en/latest/da... @fastapi.tiangolo.com #Python #FastAPI #REST

Python4DataScience reposted

python-basics-tutorial.readthedocs.ioUnicode and character encodingsSpecial characters and escape sequences:\n stands for the newline character and\t for the tab character. Character sequences that begin with a backslash and are used to represent other characters a...

21 Dec 2025

I took a look at the changes coming with Python 3.15 – and I can't wait to put them to productive use. I've already updated our tutorials: • Performance measurements: www.python4data.science/en/latest/pe... • Tachyon: www.python4data.science/en/latest/pe... #Python

python-basics-tutorial.readthedocs.iopytestpytest is an alternative to Python’s Unittest module that simplifies testing even further. pytest automatically recognises tests based on filenames and functions that start with test_, while unitte...

17 Nov 2025

We have updated the section on pytest with many exciting use cases * on command line options * on generating markers * and on parameterising exceptions python-basics-tutorial.readthedocs.io/en/latest/te... #Python #Testing #pytest

21 Oct 2025

We have updated our tutorial to data management with DVC. It also allows you to create lightweight data science and data modelling workflows and execute them in a parameterised manner: www.python4data.science/en/latest/pr... #Data #Versioncontrol #Git #DataScience #Modeling #Python

python4data.scienceConfiguring Claude Code for uvHow do we configure Claude Code to automatically use uv instead of pip for Python package management? Claude Code uses CLAUDE.md files to configure your project’s storage and context, ensuring a co...

24 Sept 2025

We have now described how to create a configuration for Claude Code so that it uses uv reliably: python4data.science/en/latest/pr... #ClaudeCode #Python #Packaging #uv

python4data.scienceConfiguring Claude Code or Cursor for uvHow do we configure Claude Code or Cursor to automatically use uv instead of pip for Python package management? Claude Code Claude Code uses CLAUDE.md files to configure your project’s storage and ...

25 Sept 2025

Now we have also described how to use uv reliably for Cursor: www.python4data.science/en/latest/pr... #CursorAI #Python #Packaging #uv