Remi's RPM repository - Blog - Mot-clé - pythonRemi's RPM repository blog Information about RPM PHP Fedora RHEL and CentOS2024-03-19T08:21:14+01:00Remi Colleturn:md5:04ca934806743f3bda12513f927cafa9DotclearAbout "dnf repomanage" performance regressionurn:md5:58960a4ffefabbed811881fa5294f8302018-01-26T07:29:00+01:002018-01-26T09:12:58+01:00RemiGeneralPHPpythonTroll<p><strong><img alt="" class="media" src="https://blog.remirepo.net/public/icons/emblem-important-2-24.png" style="{figureStyle}" /> Warning</strong>: this article can be a troll, but <strong>php</strong> is faster than <strong>python</strong>.</p> <p>Since I upgraded my workstation from Fedora 25 to <strong>Fedora 27</strong>, and thus from <a href="https://github.com/rpm-software-management/dnf">DNF</a> 1.1 to 2.7, I immediately notice than <strong>repomanage</strong>, the command used to clean old packages from my repository was much slower.</p>
<p>This week, I try to check this, and try to improve it.</p>
<h2>1. Use case</h2>
<p>I run <strong>dnf repomanage</strong> to find the 15 old RPMs in a tree with nearly 4000 source packages, keeping 5 versions of each package, so</p>
<pre>
dnf repomanage --old --keep 5 .</pre>
<h2>2. fedora 27, dnf 2.7, python 3.6</h2>
<p>The task takes about <strong>16"</strong> which is obviously too slow.</p>
<h2>3. fedora 27, php 7.1</h2>
<p>First idea was to quickly write a dirty php script to run the same task, and confirm the slowness issue</p>
<p>The first draft of this script, a 10' work, mostly parse the output of</p>
<pre>
rpm -qp --qf "%{NAME} %{EPOCHNUM} %{VERSION} %{RELEASE} %{ARCH} %{NEVR}\n" *rpm</pre>
<p>And then sort the result using the <strong>rpmdev-vercmp</strong> command</p>
<p>This is obviously a terrible and inefficient solution, especially as python have some libraries to handle rpm information in a sane way.</p>
<p>The task takes about <strong>6"</strong>.</p>
<p>Conclusion: there is a huge issue with the repomanage command, so I report it as <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1537981">bug #1537981</a> - performance issue: dnf repomanage is really slow</p>
<h2>4. rpmvercmp binding for php</h2>
<p>Just for fun, I quickly write a PHP extension providing a wrapper for <strong>librpm</strong> and exposing the single <strong>rpmvercmp</strong> function, to avoid the very bad use of an exec call in a sort callback.</p>
<pre>
$ php -r 'var_dump(rpmvercmp("1.2~RC1-1", "1.2-1"));'
int(-1)
$ php -r 'var_dump(rpmvercmp("1:1.2-1", "0:1.2-2"));'
int(1)</pre>
<p>The speed improvement was really good, the task now only takes less than a second (<strong>0.6"</strong>).</p>
<h2>5. Fedora 25, dnf 1.1, python 3.5</h2>
<p>To have a clear idea about the change, I run the same task, on the same set of packages in the Fedora 25 virtual machine.</p>
<p>And the task only takes <strong>0.27"</strong> which confirm the huge performance regression in recent version (<strong>60x slower</strong>)</p>
<h2>6. more rpmlib bindings for php</h2>
<p>As previous results with the php script was about 3x slower than the official repomanage command, I try, for fun, to improve it by adding a <strong>rpminfo</strong> function and avoid parsing the rpm command output.</p>
<pre>
$ php -n -d extension=modules/rpminfo.so -r 'print_r(rpminfo("remi-release-27.rpm"));'
Array
(
[Name] => remi-release
[Version] => 27
[Release] => 2.fc27.remi
[Arch] => noarch
)</pre>
<p>Again, nice speed improvement, the task now only takes around <strong>0.13"</strong> which is <strong>twice</strong> faster than the old python repomanage results.</p>
<h2>7. Fedora 27, yum-utils 1.1, python 2.7</h2>
<p>A quick try with the old deprecated <strong>repomanage</strong> command from the <a href="http://yum.baseurl.org/">yum</a> stack</p>
<p>Only <strong>0.26"</strong> so a bit faster than dnf, and also confirm the huge regression.</p>
<h2>8. Conclusion</h2>
<p><strong>Repomanage</strong> have a huge performance regression issue, which need to be fixed by upstream developers.</p>
<p>Running these various tests was really a funny game, and I'm happy with the result, PHP is faster than python, even on such a old "production quality" piece of code which exist in RPM distribution for years. And this is only a few hours work.</p>
<p>If you are interested in this game, you can check the code of the <strong>rpminfo</strong> extension in my git repository : <a href="https://git.remirepo.net/cgit/tools/php-rpminfo.git/">php-rpminfo</a> (see the repomanage.php script in the examples directory).</p>
<p>Perhaps I will add more bindings in this small new extension, and submit it to become an official <a href="https://pecl.php.net/">pecl</a> project, especially as the old <a href="https://pecl.php.net/package/rpmreader">rpmreader</a> extension is dead.</p>
<p>As today is <em>trollday</em>, feel free to feed the troll.</p>pdflib-lite-7.0.4-1urn:md5:68b51022b8f98ecb4e41b3a7a11168b72009-03-21T17:53:00+01:002009-04-11T11:21:35+02:00RemiArchivespdflibperlPHPpythonRPM<p>RPM of version <strong>7.0.4</strong> of this library allowing you applications to generate PDF file on the fly are availble in the <strong>remi</strong> repository and will be in <a href="http://rpmfusion.org/" hreflang="en">RPM Fusion</a> (nonfreee) after the <em>testing</em> stage</p> <p>For more information, visit the official site : <a href="http://pdflib.com/" hreflang="en">http://pdflib.com/</a>.</p>
<p><strong><img title="Attention, déc. 2008" alt="" src="https://blog.remirepo.net/public/warning.gif" /> WARNING</strong> : The <a href="http://www.pdflib.com/products/pdflib-family/pdflib-lite/pdflib-lite-licensing/" hreflang="en">License</a> restrict the use of this tool It's the reason why it is not available on official repositories (Fedora/EPEL).</p>
<p>For PHP, read the documentation on <a href="http://fr2.php.net/manual/en/ref.pdf.php" hreflang="en">PDF functions</a></p>
<p>The simplest to install is, of course, to use <a href="https://blog.remirepo.net/pages/Config-en" hreflang="en">YUM</a> !</p>
<pre>yum --enable remi install pdflib-lite</pre>
<p>Or</p>
<pre>yum --enable rpmfusion-nonfree install pdflib-lite</pre>
<p>Examples are provided by the <strong>pdflib-lite-devel</strong> sub-package.</p>
<p>If you prefer to download the RPM, search in the <a href="http://rpms.famillecollet.com/">repository</a></p>
<p>Also available : <strong>pdflib-lite-devel</strong> (C development), <strong>pdflib-lite-perl</strong> (use from perl) <strong>pdflib-lite-python</strong> (use from python) et <a href="https://blog.remirepo.net/post/2009/03/21/php-pecl-pdflib-2.1.6-2" hreflang="en">php-pecl-pdflib</a> (use from PHP).</p>
<p>I really like to use this functions in PHP which allow me to generate PDF in a simple way. Performance is really better than using others PHP classes (written in PHP) for example when converting image on the fly (TIF -> PDF). Effectivly is sometime not possible to install it on site hosted by a commercial provider.</p>pdflib-lite-7.0.4-1urn:md5:08809bb69773df332654a780d7d2bb262009-03-21T17:49:00+01:002009-04-11T11:21:01+02:00RemiArchivespdflibperlPHPpythonRPM<p>Les RPM de la version <strong>7.0.4</strong> de cette bibliothèque permettant la génération de fichiers PDF à la volée sont disponibles dans le dépôt <strong>remi</strong> et seront dans <a href="http://rpmfusion.org/" hreflang="en">RPM Fusion</a> (nonfreee) après un petit passage en <em>testing</em></p> <p>Pour plus d'information, vous pouvez consulter le site officiel : <a href="http://pdflib.com/" hreflang="en">http://pdflib.com/</a>.</p>
<p><strong><img title="Attention, déc. 2008" alt="" src="https://blog.remirepo.net/public/warning.gif" /> ATTENTION</strong> : la <a href="http://www.pdflib.com/products/pdflib-family/pdflib-lite/pdflib-lite-licensing/" hreflang="en">Licence</a> d'utilisation limite les utilisations de cet outil. C'est pour cette raison que les RPM ne seront pas disponibles dans les dépôts officiels (Fedora/EPEL).</p>
<p>Pour PHP, reportez vous à la documention sur les <a href="http://fr2.php.net/manual/fr/ref.pdf.php" hreflang="fr">Fonctions PDF</a></p>
<p>L'installation la plus simple consiste, bien sur, a utiliser <a href="https://blog.remirepo.net/pages/Config" hreflang="fr">YUM</a> !</p>
<pre>yum --enable remi install pdflib-lite</pre>
<p>Ou</p>
<pre>yum --enable rpmfusion-nonfree install pdflib-lite</pre>
<p>Les examples sont fournis dans le RPM pdflib-lite-devel.</p>
<p>Pour ceux qui préfèrent le téléchargement, voir dans <a href="http://rpms.famillecollet.com/">le dépôt</a></p>
<p>Aussi disponible : <strong>pdflib-lite-devel</strong> (développement), <strong>pdflib-lite-perl</strong> (génération en perl) <strong>pdflib-lite-python</strong> (génération en python) et <a href="https://blog.remirepo.net/post/2009/03/21/php-pecl-pdflib-2.1.6-1" hreflang="fr">php-pecl-pdflib</a> (génération en PHP).</p>
<p>J'adore utiliser ces fonctions en PHP qui me permettent trés simplement de générer des PDF. La performance est meilleure que dans les classes qui existent ailleur (génération directement en PHP) notamment lors de la conversion à la volée d'image (TIF -> PDF). Effectivement leur installation chez un hébergeur commercial n'est pas toujours possible.</p>