Pipeless project strucutre
This page provides an overview of the file and folder structure of a Pipeless project. It covers top-level files and folders, hook files and stage name conventions.
The following is an example of a generic project structure:
- pre-process.py
- process.py
- post-process.py
- pre-process.py
- process.json
- post-process.py
- pre-process.py
- process.json
- post-process.rs
Hook files
You can add any kind of file to a stage, however, the following files are reserved to be used by Pipeless:
init.{py,rs,...}: Defines the stage context, which is global for the stage. You can implement here code to setup your stage, like opening connections to external servers, loading a model from the filesystem, etc.pre-process.{py,rs,...}: Defines the pre-processing step of the stage. You can implement this file in several programming languages.post-process.{py,rs,...}: Defines the post-processing step of the stage. You can implement this file in several programming languages.process.{py,rs,...}: Defines the processing step of the stage. You can implement this file in several programming languages.process.json: Defines some metadata to use one of the supported inference runtimes in the processing step.process.jsoncannot exist on a stage folder at the same time asprocess.{py,rs,...}.
💡
We have noticed some simple use cases where you may be tempted to write all the code into a process file, however, we strongly recommend to organize your code properly according the provided hook types. This will help you with performance by leveraging the highest level of parallelization when running.
Stage folders
Stage folders can be named as you want, Pipeless uses "file-system routing" to load them.