Non-standard project setup with CoC and Pyright

Here's a quick little thing that I'll probably need to remember and might be useful to other.

At work, we have a PHP project that has automated integration tests written in Python.  The top-level directory of the Git repo is the normal PHP stuff, but the tests/integration/ directory is a Python sub-project that uses Poetry and Pytest.

Now, I use Vim as my editor and CoC with Intelliphense and Pyright for my PHp and Pythong language servers.  Since the main repo is PHP, Intelliphense works just fine.  Hwoever, Pyright needs a little help.  In particular, it didn't find the third-party dependencies or the integration test framework packages because it didn't know where to look.

Fortunately, this is easily fixed by creating a pyrightconfig.json file.  I was able to create one of those in the top-level directory of the project and add an "execution environment" to tell Pyright where to find the root of the Python project.  I set it to the "tests" directory because, while the main dir is tests/integration/,  that directory is also a Python module, so using "tests" lets Pyright find the "integration" module.

My particular file looks like this:

{
    "venvPath": "tests/integration",
    "venv": ".venv",
    "executionEnvironments": [
        {
            "root": "tests
        }
    ]
}

Random Python and WSGI tip

In light of the fact that Python 2.x loses support...now, I've been upgrading some of my personal projects that I haven't done much with recently to Python 3.  One of them is a little web-based comic book reader I call Linga (I don't remember why) that I run on my home network.

Since my home server runs Ubuntu 18.04 LTS, where Python 2.7 is the default, this has been a bit of a pain.  Before doing the conversion to Python 3.x, I had Linga set up and working using MySQL and Apache with mod_wsgi.  But once I pushed the code for the 3.x version, everything broke.  After digging in, there were several problems I needed to correct:

  1. The default mod_wsgi install uses Python 2.7, not 3.6 like I want.
  2. My WSGI script uses execfile(), which no longer exists in Python 3.x.
  3. SQLAlchemy can't load the MySQLdb module.

Luckily, the solutions to these problems weren't particularly hard to find or implement.  With a little Googling (well, DuckDuckGo-ing really, but that doesn't have the same ring to it) and a little experimentation, I found the following fixes:

  1. The version of Python to use is hard-coded into the Apache module.  So to use Python 3.6 I just needed to install the Python 3 WSGI module with apt install libapache2-mod-wsgi-py3.
  2. Apparently the recommended replacement for execfile("filename") is exec(open("filename").read()).
  3. It seems that Zen of Python's saying that "There should be one-- and preferably only one --obvious way to do it" breaks down when it comes to connecting to MySQL, as there are apparently several client libraries available.  The solution that worked for me was to pip install pymysql and run pymysql.install_as_MySQLdb() in my WSGI script.  Having to call a method to turn on MySQLdb compatibility is annoying and seems a little hacky, but it works and it's not really a big deal.

With that, Linga now works just fine.  Maybe this will save someone else having to Google the answers to these questions as well.

Flask on ICDSoft

A year or two ago, I decided to update my skill set a little and brush up on my Python. In particular, I wanted to do a little web development in Python, which I hadn't done in nearly five years. Since I wanted to start with something fairly basic, I decided to go with the Flask micro-framework. I ended up using that to build Lnto, which I've mentioned before and I swear will get its own page someday.

One big problem with this project was was that my hosting is kind of bare-bones. Don't get me wrong - the service is fine. I use ICDSoft, whom I've been with for several years. I have no complaints about the service I've received - in fact I'm quite happy with it. However, it's mostly focused on PHP and MySQL and there's no shell access. But on the other hand, I'm paying less than $5 a month with only a one-year commitment, so I don't have much to complain about.

Anyway, the problem with running Flask apps, or pretty much anything other than PHP, is that they have no documentation for that whatsoever. There's a FAQ, but it says absolutely nothing about Python other than that they "support" it. As far as I can tell, that just means that they have a Python interpreter installed and Apache CGI handler is configured to run *.py files. There's certainly no mention of running things using WSGI, which seems to be the recommended method for running most Python frameworks.

Another problem is actually installing the Flask libraries. The documentation for, well, pretty much every Python framework or app out there tells you the best way to install it is with pip or easy_install. But, of course, you need shell access to run those, assuming they're even installed on the server. (And I did check - they're not installed.)

Fortunately, getting around these problems was relatively easy using using virtualenv, which I'd nearly forgotten existe). This creates a virtual Python environment which is isolated from the rest of the system. A side-benefit of this is that virtualenv creates a copy of pip in your virtual environment.

You can use virtualenv directly from the source distribution. This was required in my scenario, since I lack any shell access, let alone root privileges. I simply extracted the virtualenv source archive, uploaded it to my host, and ran the following command (I used PHPsh, a web-based shell emulator, but copying them into a PHP script would have worked just as well):
python /path/to/virtualenv-1.11.4/virtualenv.py /path/to/myapp/venv

This create a virutal environment in the /path/to/venv directory. You can then install packages into that environment using the "activate" script to configure the shell, like this:
. /path/to/venv/bin/activate && pip install Flask

That was easy. I now have a Python environment with Flask installed. All I need do at this point is configure my application code to use it. That's accomplished with a few lines to initialized the virtualen environment and start up Flask as a CGI app:
activate_this = '/path/to/venv/bin/activate_this.py'
execfile(activate_this, dict(__file__=activate_this))
from myapp import app
from wsgiref.handlers import CGIHandler
CGIHandler().run(app)

I just re-ran that entire process using the latest version of virtualenv and it's actually quite painless.

And as a side-note, the reason I did that was because I noticed the other day that Lnto had suddenly stopped working - the server was just returning 500 errors. Which was odd because I hadn't changed anything with the app or the database in weeks. However, the answer was found on the virtualenv PyPi page:

Python bugfix releases 2.6.8, 2.7.3, 3.1.5 and 3.2.3 include a change that will cause "import random" to fail with "cannot import name urandom" on any virtualenv created on a Unix host with an earlier release of Python 2.6/2.7/3.1/3.2, if the underlying system Python is upgraded.

When I created that virtualenv environment, the server was running Python 2.6. But when I checked yesterday, the Python version was 2.7. So apparently ICDSoft upgraded their servers at some point. No big deal - just recreated the environment and I was good to go!