Learn how to create beautiful prompts in Python

Enhance your command line interface with questionary

Photo by Lukas on Unsplash

While writing my article on commitizen, I was amazed by the beauty of the command line interface (CLI) and the ease of choosing menu options using the arrow keys. It's something I've always wanted to do, having seen it in various CLIs, but alas, Click doesn't let you do it.

I dug into the commitizen code to see how it worked and that's where I discovered the questionary library. Today, we will explore what we can accomplish with this fantastic library by implementing a clone of the command poetry init. If you don’t know poetry, it is a Python package for dependency management and packaging. You can read my introduction on this topic if you want.

Just to be clear, we will not fully implement the command, things like package conflict resolution or verifying package version syntax are out of the scope of this article for simplicity purposes. You can try to implement them if you wish. 😜

Here is a preview of what we will build together.

Installation

You will need Python 3.8 or higher.

$ pip install questionary click tomkit

# or with poetry
$ poetry add questionary click tomlkit

We also add the Click dependency to create the main poet command. Tomlkit will be used to create the pyproject.toml file.

Workshop

We will start with a folder where our code will be written. By now, I have the following tree structure.

│   poetry.lock
│   pyproject.toml
│   README.md
│
├───.idea
│       .gitignore
│       workspace.xml
│
└───poet
        main.py
        __init__.py

You can use the poetry new poet command to have something similar. As you can see, I will consider here that you run your project with poetry. 🙂

I modified the pyproject.toml file to define the name of our CLI.

[tool.poetry.scripts]
poet = "poet.main:cli"

If it is hard for you to follow, you can see the final project on my GitHub.

The main.py file will look like this at the beginning.

import click


@click.version_option('0.1.0', message='%(prog)s version %(version)s')
@click.group(context_settings={'help_option_names': ['-h', '--help']})
def cli():
    """Dummy command to mimic "poetry init" command."""

Now, we will create subpackage commands holding all the subcommands to the main poet command. In this package, there will be an init.py module that represents the poetry init command. Just start with a hello world to be sure everything is correctly set up.

import click


@click.command()
def init():
    """Initialize the pyproject.toml with correct metadata."""
    click.echo('Hello World!')

Now we will modify the main.py module to add the subcommand.

import click

from poet.commands.init import init


@click.version_option('0.1.0', message='%(prog)s version %(version)s')
@click.group(context_settings={'help_option_names': ['-h', '--help']})
def cli():
    """Dummy command to mimic "poetry init" command."""


cli.add_command(init)

You should run poetry install to install the project in editable mode. Now you can type poet init in the terminal and you should see the Hello World! message.

Now let’s start the fun part! Create a module utils.py that will hold subprocess and prompt utilities near the main.py module. We will wrap each prompt in a more meaningful function. We will start with the prompt asking for the package name.

from pathlib import Path

import questionary


def ask_name() -> questionary.Question:
    return questionary.text('Package name', default=Path.cwd().name)

The most basic function in the questionary library is the text one. Its usage looks like the built-in input function. But in this case, it returns a Question object that we will use in the init.py module.

We also provide a default value to the prompt that represents the current working directory.

Now in init.py, let’s write this.

import click
import questionary

from poet.utils import ask_name


@click.command()
def init():
    """Initialize the pyproject.toml with correct metadata."""
    form = questionary.form(name=ask_name()).ask()
    click.echo(form)

The questionary form function returns a Form used to prompt the user. The ask method will collect all the values for the different prompts used through the form. At the end, we will have the values filed in the form variable.

Now, try to run the init command, you should see something similar to the following:

poet init
? Package name: poet

The default name is shown, you can type Enter if you want to keep it or replace it with the name of your choice.

Perfect! We can now implement the second prompt to ask for the package version. It is similar to our first helper function.

def ask_version() -> questionary.Question:
    return questionary.text('Version:', default='0.1.0')

Now, we can modify the init.py module.

...
from poet.utils import ask_name, ask_version


@click.command()
def init():
    """Initialize the pyproject.toml with correct metadata."""
    form = questionary.form(name=ask_name(), version=ask_version()).ask()
    click.echo(form)

You can test to confirm everything is OK. The third information is similar.

# utils.py
def ask_description() -> questionary.Question:
    return questionary.text('Description:')

# init.py
@click.command()
def init():
    """Initialize the pyproject.toml with correct metadata."""
    form = questionary.form(name=ask_name(), version=ask_version(), description=ask_description()).ask()
    click.echo(form)

For the author's information, we will show how to validate input. Let’s say we want the input to be in the form name <email>. We will need to install the email-validator library because validating emails is hard. 😆

But before validating the author data, let’s see how we can get default information from Git.

import subprocess


def run_process(arguments: list) -> str:
    try:
        result = subprocess.run(arguments, capture_output=True, text=True)
        return result.stdout.strip()
    except subprocess.CalledProcessError:
        return ''


def get_default_author_information() -> str:
    name = run_process(['git', 'config', '--global', 'user.name'])
    email = run_process(['git', 'config', '--global', 'user.email'])
    if not name and not email:
        return ''
    return f'{name} <{email}>'

We use the subprocess module to run the following Git commands:

$ git config --global user.name
$ git config --global user.email

Now let’s look at the validation code. It may be daunting at first sight, but it is not that complicated, trust me. 😉

class AuthorValidator(questionary.Validator):
    @staticmethod
    def _check_composition(author_data: str, parts: list[str]) -> None:
        if len(parts) < 2:
            raise questionary.ValidationError(message='missing email information', cursor_position=len(author_data))

    @staticmethod
    def _get_cursor_position(author_data: str, author_name: str) -> int:
        # it is a best-effort attempt to get the position of where the email starts.
        # - in the best scenario, the format is respected and the email is not correct, so we search "<" in the data
        # - if not we compute the author name and add 2 to consider the space between the author name and the email
        try:
            return author_data.index('<')
        except ValueError:
            return len(author_name) + 2

    @staticmethod
    def _check_email_tag(email: str, cursor_position: int) -> None:
        if not email.startswith('<') or not email.endswith('>'):
            raise questionary.ValidationError(
                message='email must be in the form <EMAIL>', cursor_position=cursor_position
            )

    @staticmethod
    def _check_email_format(email: str, cursor_position: int) -> None:
        try:
            # the second argument is to avoid a DNS check
            email_validator.validate_email(email[1 :-1], check_deliverability=False)
        except email_validator.EmailNotValidError as e:
            raise questionary.ValidationError(message=str(e), cursor_position=cursor_position)

    def _check_author_data_is_correct(self, author_data: str) -> None:
        parts = author_data.split()
        self._check_composition(author_data, parts)

        author_name = ' '.join(parts[:-1])
        email = parts[-1]
        cursor_position = self._get_cursor_position(author_data, author_name)
        self._check_email_tag(email, cursor_position)
        self._check_email_format(email, cursor_position)

    def validate(self, document: Document) -> None:
        author_data = document.text.strip()
        # author information is not mandatory
        if not author_data:
            return

        # but if present, we need to validate it
        self._check_author_data_is_correct(author_data)


def ask_author_information() -> questionary.Question:
    return questionary.text('Author:', default=get_default_author_information(), validate=AuthorValidator)

The most basic form of a questionary validator looks like this:

class NameValidator(questionary.Validator):
    def validate(self, document: Document) -> None:
        if len(document.text) == 0:
            raise questionary.ValidationError(message='name must not be empty', cursor_position=0)

We inherit the questionary Validator class and we need to implement the validate method. The cursor position is just a hint to show our users where the error happened, it is not mandatory. So if we look at the validation method in our class implementing author information validation, it does two things:

• Check if the author's information is empty, if this is the case, we skip further validation because this information can be empty.

• If not is not the case, we want to make sure the author’s information respects the form name <email>. This is what checks the method self._check_author_data_is_correct(author_data).

The latter method calls three inner methods to verify that:

• The email information is present

• The email starts with < and ends with >.

• The email is correct by calling email_validator.validate_email.

You also have another helper to compute the cursor position. We combine all of this in a helper function that will be exported. Note that we don’t instantiate the validator, we just pass a reference of the class.

def ask_author_information() -> questionary.Question:
    return questionary.text('Author:', default=get_default_author_information(), validate=AuthorValidator)

Now modify your init.py module.

@click.command()
def init():
    """Initialize the pyproject.toml with correct metadata."""
    form = questionary.form(
        name=ask_name(), version=ask_version(), description=ask_description(), author=ask_author_information()
    ).ask()
    click.echo(form)

I encourage you to test different error cases to see how the prompt reacts and print helpful messages.

For the license information, it will also be a free text but this time, we will add auto-completion. Let’s see the code.

def ask_license_information() -> questionary.Question:
    # information taken from https://choosealicense.com/ (apart the last one)
    metadata = {
        'MIT': 'A short and simple permissive license with conditions only requiring preservation of copyright and'
        ' license notices.',
        'Apache 2.0': 'A permissive license whose main conditions require preservation of copyright and '
        'license notices.',
        'GPLv2': 'The GNU GPL is the most widely used free software license and has a strong copyleft requirement. '
        'When distributing derived works, the source code of the work must be made available '
        'under the same license.',
        'GPLv3': 'Permissions of this strong copyleft license are conditioned on making available complete source'
        ' code of licensed works and modifications, which include larger works using a licensed work, '
        'under the same license',
        'Unlicense': 'A license with no conditions whatsoever which dedicates works to the public domain.',
        'Proprietary': 'Proprietary License',
    }
    return questionary.autocomplete('License:', choices=list(metadata.keys()), metadata=metadata)

We create a dictionary where the keys are valid licenses and the values are descriptions of these licenses. The autocomplete function takes a list of choices that will trigger auto-completion. Update the init.py module.

@click.command()
def init():
    """Initialize the pyproject.toml with correct metadata."""
    form = questionary.form(
        name=ask_name(),
        version=ask_version(),
        description=ask_description(),
        author=ask_author_information(),
        license=ask_license_information(),
    ).ask()
    click.echo(form)

You can test it to see how the auto-completion works. Be careful to use the down arrow key to select an autocomplete suggestion. 😉

For the readme information, we will use a select form.

# utils.py
def ask_readme_file_type() -> questionary.Question:
    return questionary.rawselect('README style:', choices=['README.md', 'README.rst', 'README.txt'])

# init.py
@click.command()
def init():
    """Initialize the pyproject.toml with correct metadata."""
    form = questionary.form(
        name=ask_name(),
        version=ask_version(),
        description=ask_description(),
        author=ask_author_information(),
        license=ask_license_information(),
        readme=ask_readme_file_type(),
    ).ask()
    click.echo(form)

Questionary has two APIs to deal with choices option, select and rawselect. The latter has the advantage that you can use shortcuts to specify a choice. It also implies that the list of choices is relatively small (less than 10 elements). 🙂 We will use the first select option below, don’t worry.

Now for the Python version, we will add validation to support only major versions 2 or 3 of the language. This is not really what poetry expects, but for simplicity, we will just consider a minimum Python version to use.

# utils.py

...
import sys

def get_default_python_version() -> str:
    return f'{sys.version_info.major}.{sys.version_info.minor}'


class PythonVersionValidator(questionary.Validator):
    def validate(self, document: Document) -> None:
        pattern = r'^[23]\.\d+'
        if re.match(pattern, document.text) is None:
            raise questionary.ValidationError(
                message='The python version must be in the form 2.X or 3.X where X represents an integer.'
            )


def ask_python_version() -> questionary.Question:
    return questionary.text('Python Version:', default=get_default_python_version(), validate=PythonVersionValidator)

# init.py

@click.command()
def init():
    """Initialize the pyproject.toml with correct metadata."""
    form = questionary.form(
        name=ask_name(),
        version=ask_version(),
        description=ask_description(),
        author=ask_author_information(),
        license=ask_license_information(),
        readme=ask_readme_file_type(),
        python=ask_python_version(),
    ).ask()
    click.echo(form)

Ok, we will stop there with the form because the following information we are going to ask our users will need loops to add package dependencies, so the questionary form is not adequate. First of all, let’s see how we can fetch package information from PyPI.

I didn’t see a way to get PyPI packages using a pattern with the official PyPI JSON API. The best we can do is to get the metadata from a package. It seems there exists an old legacy XML-RPC API capable of that but it is now deprecated. So I decided to scrape data on the official website. We need to install two new libraries for that purpose.

$ poetry add httpx parsel

• httpx is for me the successor of requests. It is well maintained in contrast with the former and has asynchronous support via anyio.

• parsel is used to parse HTML/XML/JSON documents. It is the foundation of the well-known scraping library scrapy.

If we look at the HTML structure of a page like https://pypi.org/search/?q=pytest, we notice that the key information of each package has the following structure:

<!-- HTML struture of the first package listed: pytest123 -->

<a class="package-snippet" href="/project/pytest123/">
    <h3 class="package-snippet__title">
      <span class="package-snippet__name">pytest123</span>
      <span class="package-snippet__version">0.0.1</span>
      <span class="package-snippet__created"><time datetime="2016-06-06T00:59:43+0000" data-controller="localized-time" data-localized-time-relative="true" data-localized-time-show-time="false">
  6 juin 2016
</time></span>
    </h3>
    <p class="package-snippet__description">Test Library</p>
  </a>
</li>

Therefore it is easy to parse the relevant information. Create a scraper.py module near the utils.py module with the following content:

import parsel
import httpx


def find_relevant_packages(package: str) -> dict[str, str]:
    packages = {}
    response = httpx.get('https://pypi.org/search/', params={'q': package})
    if response.status_code >= 400:
        return packages

    selector = parsel.Selector(text=response.text)
    h3_list = selector.xpath('//h3[@class="package-snippet__title"]')
    for h3 in h3_list:
        version = h3.xpath('span[@class="package-snippet__version"]/text()').get()
        name = h3.xpath('span[@class="package-snippet__name"]/text()').get()
        packages[name] = version

    return packages

I'm not going to go into detail on how to use XPath here, but for those who want more details, I recommend the course I took myself to learn how to use it. 😁

Now in utils.py add the following functions:

...
from poet.scraper import find_relevant_packages


def get_first_ten_packages(packages: dict[str, str]) -> dict[str, str]:
    filtered_items = list(packages.items())[:10]
    return dict(filtered_items)


def get_formatted_package_version(packages: dict[str, str], package: str, package_version: str) -> str:
    if not package_version:
        return f'^{packages[package]}'

    return package_version


def ask_dependencies(dependency_type: str) -> dict:
    dependencies = {}
    user_wants_to_continue = questionary.confirm(
        f'\nWould you like to define your {dependency_type} dependencies interactively?', default=True
    ).ask()
    while user_wants_to_continue:
        package = questionary.text('Add a package (leave blank to skip):').ask()
        if not package:
            break
        packages = find_relevant_packages(package)
        packages_length = len(packages)
        click.echo(f'Found {packages_length} packages matching {package}')
        if packages_length > 10:
            click.echo('Showing the first 10 matches')
            packages = get_first_ten_packages(packages)

        selected_package = questionary.select('Select the package to add', choices=list(packages.keys())).ask()
        package_version = questionary.text(
            'Enter the version constraint to require (or leave blank to use the latest version):'
        ).ask()
        package_version = get_formatted_package_version(packages, selected_package, package_version)

        dependencies[selected_package] = package_version
        click.echo(f'Using version {package_version} for {selected_package}\n')

    return dependencies


def ask_main_dependencies() -> dict:
    return ask_dependencies('main')


def ask_development_dependencies() -> dict:
    return ask_dependencies('development')

The main function here is ask_dependencies. Here is how it works.

It takes a single argument, a name specifying the type of dependencies we want to install. It can be the main or development dependencies of the project. Let’s dissect the code.

dependencies = {}
user_wants_to_continue = questionary.confirm(
    f'\nWould you like to define your {dependency_type} dependencies interactively?', default=True
).ask()
while user_wants_to_continue:
    ...

return dependencies

In the previous snippet, we ask the user if he wants to define the dependencies interactively, if not, we stop here and return an empty dictionary.

package = questionary.text('Add a package (leave blank to skip):').ask()
if not package:
	break
packages = find_relevant_packages(package)
packages_length = len(packages)
click.echo(f'Found {packages_length} packages matching {package}')
if packages_length > 10:
    click.echo('Showing the first 10 matches')
    packages = get_first_ten_packages(packages)

We ask the user for the package name he wants. If the answer is empty, it means we want to end there. If not, we use our scraper function to find relevant packages. If we have more than 10 possibilities, we only print the first 10 packages and inform the user of the number of possibilities available, this is what poetry does.

selected_package = questionary.select('Select the package to add', choices=list(packages.keys())).ask()
package_version = questionary.text(
    'Enter the version constraint to require (or leave blank to use the latest version):'
).ask()
package_version = get_formatted_package_version(packages, selected_package, package_version)

dependencies[selected_package] = package_version
click.echo(f'Using version {package_version} for {selected_package}\n')

We use the questionary select feature to list the packages. Once the user made his choice, we ask another question to ask for the package version. Once it is done, we use a small function to add a circumflex character if needed in front of the package version like poetry does. We register the package and the loop continues with the next package until the user is satisfied and enters an empty value to leave the loop.

The process is done two times for the main and development dependencies. This is why we have the last two functions.

def ask_main_dependencies() -> dict:
    return ask_dependencies('main')


def ask_development_dependencies() -> dict:
    return ask_dependencies('development')

Now in the init.py module, use the latter functions we just defined (don’t forget to import them).

...
@click.command()
def init():
    """Initialize the pyproject.toml with correct metadata."""
    form = questionary.form(
        name=ask_name(),
        version=ask_version(),
        description=ask_description(),
        author=ask_author_information(),
        license=ask_license_information(),
        readme=ask_readme_file_type(),
        python=ask_python_version(),
    ).ask()
    click.echo(form)
    main_dependencies = ask_main_dependencies()
    click.echo(main_dependencies)
    development_dependencies = ask_development_dependencies()
    click.echo(development_dependencies)

You can play with it to see how it works.

Now the final piece of our project is to preview the pyproject.toml file that our command will create and ask the user if he wants to save it. The first function we will add in the utils.py module is one to create a dictionary of all the data needed by poetry.

def construct_pyproject_file(
    form: dict[str, str], main_dependencies: dict[str, str], development_dependencies: dict[str, str]
) -> dict[str, Any]:
    python_version = form.pop('python')
    author = form.pop('author')
    poetry_dict = {
        'tool': {
            'poetry': {
                'name': form['name'],
                'version': '0.1.0',
                'description': form['description'],
                'authors': [author],
                'license': form['license'],
                'readme': form['readme'],
                'dependencies': {
                    'python': python_version,
                    **main_dependencies,
                },
                'group': {'dev': {'dependencies': development_dependencies}},
            }
        },
        'build-system': {'requires': ['poetry-core'], 'build-backend': 'poetry.core.masonry.api'},
    }

    return poetry_dict

If you look at the dictionary structure, it should be familiar to what you see in a classic pyproject.toml file. In fact, I just loaded a dictionary from one of my poetry projects using the tomlkit parse utility and I replaced some information from what I got from the user. 🙃

Here you see that the function takes three arguments:

• The first one is the questionary form object

• The second is the dictionary of the main dependencies

• The last is the dictionary of development dependencies

We just arrange information in the correct order and return the final dictionary.

Now we need a function to display an eventually pyproject.toml file. The following function takes a single argument, a string resulting from the poetry dictionary transformed using the tomlkit dumps feature (we will use it sooner).

def preview_pyproject_file(pyproject_data: str) -> None:
    click.echo('Generated file\n')
    click.echo(pyproject_data)

If the user wants to save the file, we need a function to do that.

def generate_pyproject_file(pyproject_data: str) -> None:
    with open('pyproject.toml', 'w') as f:
        f.write(pyproject_data)

Now, we assemble all the logic to handle the pyproject.toml file in the following function.

def handle_pyproject_file_creation(
    form: dict[str, str], main_dependencies: dict[str, str], development_dependencies: dict[str, str]
):
    poetry_dict = construct_pyproject_file(form, main_dependencies, development_dependencies)
    pyproject_data = tomlkit.dumps(poetry_dict)
    preview_pyproject_file(pyproject_data)

    user_wants_to_create_file = questionary.confirm('Do you confirm generation?', default=True).ask()
    if user_wants_to_create_file:
        generate_pyproject_file(pyproject_data)
        click.secho('The pyproject toml file was generated!', fg='green')

You can see the usage of the tomlkit dumps method before previewing the file.

The final code in the init.py module looks like this.

import click
import questionary

from poet.utils import (
    ask_name,
    ask_version,
    ask_description,
    ask_author_information,
    ask_license_information,
    ask_python_version,
    ask_readme_file_type,
    ask_main_dependencies,
    ask_development_dependencies,
    handle_pyproject_file_creation
)


@click.command()
def init():
    """Initialize the pyproject.toml with correct metadata."""
    form = questionary.form(
        name=ask_name(),
        version=ask_version(),
        description=ask_description(),
        author=ask_author_information(),
        license=ask_license_information(),
        readme=ask_readme_file_type(),
        python=ask_python_version(),
    ).ask()
    main_dependencies = ask_main_dependencies()
    development_dependencies = ask_development_dependencies()
    handle_pyproject_file_creation(form, main_dependencies, development_dependencies)

Congratulations if you follow till the end! You did well!

As I said before, the final code is on GitHub. I refactored a bit the utils.py module to make it a package with small modules inside of it.

This is all for this article, hope you enjoy reading it. Take care of yourself and see you soon. 😁