Getting started#
We’re pleased that you are interested in working on Warehouse.
Your first pull request#
After you set up your development environment and ensure you can run the tests and build the documentation (using the instructions in this document), take a look at our guide to the Warehouse codebase. Then, look at our open issues that are labelled “good first issue”, find one you want to work on, comment on it to say you’re working on it, then submit a pull request. Use our Submitting patches documentation to help.
Setting up a development environment to work on Warehouse should be a straightforward process. If you have any difficulty, contact us so we can improve the process:
For bug reports or general problems, file an issue on GitHub;
For real-time chat with other PyPA developers, join
#pypa-dev
on Libera, or the PyPA Discord;For longer-form questions or discussion, visit Discourse.
Detailed installation instructions#
Getting the Warehouse source code#
Fork the repository on GitHub and clone it to your local machine:
git clone git@github.com:YOUR-USERNAME/warehouse.git
Add a remote and regularly sync to make sure you stay up-to-date with our repository:
git remote add upstream https://github.com/pypi/warehouse.git
git checkout main
git fetch upstream
git merge upstream/main
Configure the development environment#
Note
In case you are used to using a virtual environment for Python development: it’s unnecessary for Warehouse development. Our Makefile scripts execute all developer actions inside Docker containers.
Why Docker?#
Docker simplifies development environment set up.
Warehouse uses Docker and Docker Compose
to automate setting up a “batteries included” development environment. The
Dockerfile
and docker-compose.yml
files include all the
required steps for installing and configuring all the required external
services of the development environment.
Installing Docker#
Install Docker Engine
The best experience for building Warehouse on Windows 10 is to use the Windows Subsystem for Linux (WSL) in combination with both Docker for Windows and Docker for Linux. Follow the instructions for both platforms, and see Docker and Windows Subsystem for Linux Quirks for extra configuration instructions.
Verifying Docker installation#
Check that Docker is installed: docker -v
Install Docker Compose#
Install Docker Compose using the Docker-provided installation instructions.
Note
Docker Compose will be installed by Docker for Mac and Docker for Windows automatically.
Verifying Docker Compose installation#
Check that Docker Compose is installed: docker compose version
Verifying the necessary ports are available#
Warehouse needs access to a few local ports in order to run, namely ports
80
, 5433
, and 9000
. You should check each of these for availability
with the lsof
command.
For example, checking port 80
:
sudo lsof -i:80 | grep LISTEN
If the port is in use, the command will produce output, and you will need to determine what is occupying the port and shut down the corresponding service. Otherwise, the port is available for Warehouse to use, and you can continue.
Alternately, you may set the WEB_HOST
environment variable for
docker compose
to use instead. An example:
export WEB_HOST=8080
make ...
# or inline:
WEB_HOST=8080 make ...
Building the Warehouse Container#
Once you have Docker and Docker Compose installed, run:
make build
in the repository root directory.
This will pull down all of the required docker containers, build Warehouse and
run all of the needed services. The Warehouse repository will be mounted inside
the Docker container at /opt/warehouse/src/
. After the initial build,
you should not have to run this command again.
Running the Warehouse container and services#
You have to start the Docker services that make up the Warehouse application.
Tip
These services need ~4 GB of RAM dedicated to Docker to work. This is more than the default setting of the Docker Engine of 2 GB. Thus, you need to increase the memory allocated to Docker in Docker Preferences (on Mac) or Docker Settings (on Windows) by moving the slider to 4 GB in the GUI.
If you are using Linux, you may need to configure the maximum map count to get the elasticsearch up and running. According to the documentation this can be set temporarily:
# sysctl -w vm.max_map_count=262144
or permanently by modifying the vm.max_map_count
setting in your
/etc/sysctl.conf
.
Also check that you have more than 5% disk space free, otherwise
elasticsearch will become read only. See flood_stage
in the
elasticsearch disk allocation docs.
Once make build
has finished, run the command:
make initdb
This command will:
create a new Postgres database,
install example data to the Postgres database,
run migrations,
load some example data from Test PyPI, and
index all the data for the search database.
Once the make initdb
command has finished, you are ready to continue:
make serve
This command starts the containers that run Warehouse on your local machine. After the initial build process, you will only need this command each time you want to startup Warehouse locally.
make serve
will produce output for a while, and will not exit. Eventually
the output will cease, and you will see a log message indicating that either
the web
service has started listening:
warehouse-web-1 | [2022-12-26 19:27:12 +0000] [1] [INFO] Starting gunicorn 20.1.0
warehouse-web-1 | [2022-12-26 19:27:12 +0000] [1] [INFO] Listening at: http://0.0.0.0:8000 (1)
warehouse-web-1 | [2022-12-26 19:27:12 +0000] [1] [INFO] Using worker: sync
warehouse-web-1 | [2022-12-26 19:27:12 +0000] [7] [INFO] Booting worker with pid: 7
or that the static
container has finished compiling the static assets:
warehouse-static-1 |
warehouse-static-1 | webpack 5.75.0 compiled with 1 warning in 6610 ms
or maybe something else.
Viewing Warehouse in a browser#
At this point all the services are up, and web container is listening on port 80. It’s accessible at http://localhost:80/.
Note
If you are using docker-machine
on an older version of macOS or
Windows, the warehouse application might be accessible at
https://<docker-ip>:80/
instead. You can get information about the
docker container with docker-machine env
Note
On Firefox, the logos might show up as black rectangles due to the Content Security Policy used and an implementation bug in Firefox (see this bug report for more info).
If you’ve set a different port via the WEB_HOST
environment variable,
use that port instead.
Logging in to Warehouse#
In the development environment, the password for every account has been set to
the string password
. You can log in as any account at
http://localhost:80/account/login/.
To log in as an admin user, log in as ewdurbin
with the password
password
.
Stopping Warehouse and other services#
In the terminal where make serve
is running, you can use Control-C
to gracefully stop all Docker containers, and thus the one running the
Warehouse application.
Or, from another terminal, use make stop
in the Warehouse
repository root; that’ll stop all the Docker processes with
warehouse
in the name.
What did we just do and what is happening behind the scenes?#
The repository is exposed inside of the web container at
/opt/warehouse/src/
and Warehouse will automatically reload
when it detects any changes made to the code.
The example data located in dev/example.sql.xz
is taken from
Test PyPI and has been sanitized to remove anything private.
Running your developer environment after initial setup#
You won’t have to initialize the database after the first time you do
so, and you will rarely have to re-run make build
. Ordinarily, to
access your developer environment, you’ll:
make serve
View Warehouse in the browser at http://localhost:80/.
Debugging the webserver#
If you would like to use a debugger like pdb that allows you to drop
into a shell, you can use make debug
instead of make serve
.
Troubleshooting#
Errors when executing make build
#
If you are using Ubuntu and
invalid reference format
error is displayed, you can fix it by installing Docker through Snap.snap install docker
Errors when executing make serve
#
If the
Dockerfile
is edited or new dependencies are added (either by you or a prior pull request), a new container will need to built. A new container can be built by runningmake build
. This should be done before runningmake serve
again.If
make serve
hangs after a new build, you should stop any running containers and repeatmake serve
.To run Warehouse behind a proxy set the appropriate proxy settings in the
Dockerfile
.If
sqlalchemy.exec.OperationalError
is displayed inlocalhost
aftermake serve
has been executed, shut down the Docker containers. When the containers have shut down, runmake serve
in one terminal window while runningmake initdb
in a separate terminal window.
Errors when executing make purge
#
If
make purge
fails with a permission error, check ownership and permissions onwarehouse/static
.docker compose
is spawning containers with docker. Generally on Linux that process is running as root. So when it writes files back to the file system as the static container does those are owned by root. So your docker daemon would be running as root, so your user doesn’t have permission to remove the files written by the containers.sudo make purge
will work.
Errors when executing make initdb
#
If
make initdb
fails with a timeout like:urllib3.exceptions.ConnectTimeoutError: (<urllib3.connection.HTTPConnection object at 0x8beca733c3c8>, 'Connection to elasticsearch timed out. (connect timeout=30)')
you might need to increase the amount of memory allocated to docker, since elasticsearch wants a lot of memory (Dustin gives warehouse ~4GB locally). Refer to the tip under Running the Warehouse container and services section for more details.
“no space left on device” when using docker compose
#
docker compose may leave orphaned volumes during teardown. If you run into the message “no space left on device”, try running the following command (assuming Docker >= 1.9):
docker volume rm $(docker volume ls -qf dangling=true)
Note
This will delete orphaned volumes as well as directories that are not
volumes in /var/lib/docker/volumes
(Solution found and further details available at https://github.com/chadoe/docker-cleanup-volumes)
make initdb
is slow or appears to make no progress#
This typically occur when Docker is not allocated enough memory to perform the
migrations. Try modifying your Docker configuration to allow more RAM for each
container, temporarily stop make_serve
and run make initdb
again.
make initdb
complains about PostgreSQL Version#
You built a Warehouse install some time ago and PostgreSQL has been updated.
If you do not need the data in your databases, it might be best to just blow
away your builds + docker
containers and start again:
make purge
docker volume rm $(docker volume ls -q --filter dangling=true)
Compilation errors in non-Docker development#
While Warehouse is designed to be developed in using Docker, you may have tried to install Warehouse’s requirements in your system or virtual environment. This is discouraged as it can result in compilation errors due to your system not including libraries or binaries required by some of Warehouse’s dependencies.
An example of such dependency is psycopg2 which requires PostgreSQL binaries and will fail if not present.
If there’s a specific use case you think requires development outside Docker please raise an issue in Warehouse’s issue tracker.
Disabling services locally#
Some services, such as Elasticsearch, consume a lot of resources when running locally, but might not always be necessary when doing local development.
To disable these locally, you can create a docker-compose.override.yaml
file to override any settings in the docker-compose.yaml
file. To
individually disable services, modify their entrypoint to do something else:
version: "3"
services:
elasticsearch:
entrypoint: ["echo", "Elasticsearch disabled"]
Note that disabling services might cause things to fail in unexpected ways.
This file is ignored in Warehouse’s .gitignore
file, so it’s safe to keep
in the root of your local repo.
Docker and Windows Subsystem for Linux Quirks#
Once you have installed Docker for Windows, the Windows Subsystem for Linux, and Docker and Docker Compose in WSL, there are some extra configuration steps to deal with current quirks in WSL. Nick Janetakis has a detailed blog post on these steps, including installation, but this is a summary of the required steps:
1. In WSL, run sudo mkdir /c
and sudo mount --bind /mnt/c /c
to mount your root drive at /c
(or whichever drive you are
using). You should clone into this mount and run
docker compose from within it, to ensure that when volumes
are linked into the container they can be found by Hyper-V.
2. In Windows, configure Docker to enable “Expose daemon on
tcp://localhost:2375
without TLS”. Note that this may expose your
machine to certain remote code execution attacks, so use with
caution.
3. Add export DOCKER_HOST=tcp://0.0.0.0:2375
to your
.bashrc
file in WSL, and/or run it directly to enable for the
current session. Without this, the docker command in WSL
will not be able to find the daemon running in Windows.
Building Styles#
Styles are written in the scss variant of Sass and compiled using
webpack. They will be automatically built when changed when
make serve
is running.
Running the Interactive Shell#
There is an interactive shell available in Warehouse which will automatically configure Warehouse and create a database session and make them available as variables in the interactive shell.
To run the interactive shell, simply run:
make shell
The interactive shell will have the following variables defined in it:
config |
The Pyramid |
db |
The SQLAlchemy ORM |
To use the db
object in the interactive shell, import the class you’re
planning to use. For example, if I wanted to use the User object, I would
do this:
$ make shell
docker compose run --rm web python -m warehouse shell
Starting warehouse_redis_1 ...
...
(InteractiveConsole)
>>>
>>> from warehouse.accounts.models import User
>>> db.query(User).filter_by(username='test').all()
[User(username='test')]
You can also run the IPython shell as the interactive shell. To do so export
the environment variable WAREHOUSE_IPYTHON_SHELL prior to running the
make build
step:
export WAREHOUSE_IPYTHON_SHELL=1
Now you will be able to run the make shell
command to get the IPython
shell.
Running tests and linters#
Note
PostgreSQL 9.4 is required because of pgcrypto
extension
The Warehouse tests are found in the tests/
directory and are
designed to be run using make.
To run all tests, in the root of the repository:
make tests
make static_tests
This will run the tests with the supported interpreter as well as all of the additional testing that we require.
If you want to run a specific test, you can use the T
variable:
T=tests/unit/i18n/test_filters.py make tests
You can run linters, programs that check the code, with:
make lint
Warehouse uses black for opinionated formatting and linting. You can reformat with:
make reformat
Building documentation#
The Warehouse documentation is stored in the docs/
directory. It is written in reStructured Text and rendered using
Sphinx.
Use make to build the documentation. For example:
make docs
The HTML documentation index can now be found at
docs/_build/html/index.html
.
Building the docs requires Python 3.8. If it is not installed, the make command will give the following error message:
make: python3.8: Command not found
Makefile:53: recipe for target '.state/env/pyvenv.cfg' failed
make: *** [.state/env/pyvenv.cfg] Error 127
Building translations#
Warehouse is translated into a number of different locales, which are stored in
the warehouse/locale/
directory.
These translation files contain references to untranslated text in source code and HTML templates, as well as the translations which have been submitted by our volunteer translators.
When making changes to files with strings marked for translation, it’s necessary to update these references any time source strings are change, or the line numbers of the source strings in the source files.
Use make to build the translations. For example:
make translations
What next?#
Look at our open issues that are labelled “good first issue”, find one you want to work on, comment on it to say you’re working on it, then submit a pull request. Use our Submitting patches documentation to help.
Talk with us#
You can find us via a GitHub issue, #pypa
or #pypa-dev
on
Libera, the PyPA Discord or Discourse, to ask questions or get
involved. And you can meet us in person at packaging sprints.
Learn about Warehouse and packaging#
Resources to help you learn Warehouse’s context: