Plantage à intervalle "presque" régulière

Bonjour,

C’est après plusieurs semaines que je viens désespérément poster sur ce forum afin de trouver de l’aide concernant le FabManager d’une communauté de communes qui, sans raison purement apparente, est inaccessible à intervalle presque régulière (tous les 3 jours, des fois tous les 2 jours, des fois une fois par semaine).

D’après ce que je comprends des logs lors du plantage, les containers essayent de démarrer alors que ceux-ci sont déjà up (logfile app-stderr.log)

A server is already running. Check /usr/src/app/tmp/pids/server.pid. Your RubyGems version (3.0.3.1) has a bug that prevents required_ruby_versionfrom working for Bundler. Any scripts that usegem install bundlerwill break as soon as Bundler drops support for your Ruby version. Please upgrade RubyGems to avoid future breakage and silence this warning by runninggem update --system 3.2.3A server is already running. Check /usr/src/app/tmp/pids/server.pid. Your RubyGems version (3.0.3.1) has a bug that preventsrequired_ruby_versionfrom working for Bundler. Any scripts that usegem install bundlerwill break as soon as Bundler drops support for your Ruby version. Please upgrade RubyGems to avoid future breakage and silence this warning by runninggem update --system 3.2.3A server is already running. Check /usr/src/app/tmp/pids/server.pid. Your RubyGems version (3.0.3.1) has a bug that preventsrequired_ruby_versionfrom working for Bundler. Any scripts that usegem install bundlerwill break as soon as Bundler drops support for your Ruby version. Please upgrade RubyGems to avoid future breakage and silence this warning by runninggem update --system 3.2.3A server is already running. Check /usr/src/app/tmp/pids/server.pid. Your RubyGems version (3.0.3.1) has a bug that preventsrequired_ruby_versionfrom working for Bundler. Any scripts that usegem install bundlerwill break as soon as Bundler drops support for your Ruby version. Please upgrade RubyGems to avoid future breakage and silence this warning by runninggem update --system 3.2.3A server is already running. Check /usr/src/app/tmp/pids/server.pid. Your RubyGems version (3.0.3.1) has a bug that preventsrequired_ruby_versionfrom working for Bundler. Any scripts that usegem install bundlerwill break as soon as Bundler drops support for your Ruby version. Please upgrade RubyGems to avoid future breakage and silence this warning by runninggem update --system 3.2.3A server is already running. Check /usr/src/app/tmp/pids/server.pid. Your RubyGems version (3.0.3.1) has a bug that preventsrequired_ruby_versionfrom working for Bundler. Any scripts that usegem install bundlerwill break as soon as Bundler drops support for your Ruby version. Please upgrade RubyGems to avoid future breakage and silence this warning by runninggem update --system 3.2.3A server is already running. Check /usr/src/app/tmp/pids/server.pid. Your RubyGems version (3.0.3.1) has a bug that preventsrequired_ruby_versionfrom working for Bundler. Any scripts that usegem install bundlerwill break as soon as Bundler drops support for your Ruby version. Please upgrade RubyGems to avoid future breakage and silence this warning by runninggem update --system 3.2.3A server is already running. Check /usr/src/app/tmp/pids/server.pid. Your RubyGems version (3.0.3.1) has a bug that preventsrequired_ruby_versionfrom working for Bundler. Any scripts that usegem install bundlerwill break as soon as Bundler drops support for your Ruby version. Please upgrade RubyGems to avoid future breakage and silence this warning by runninggem update --system 3.2.3A server is already running. Check /usr/src/app/tmp/pids/server.pid. Your RubyGems version (3.0.3.1) has a bug that preventsrequired_ruby_versionfrom working for Bundler. Any scripts that usegem install bundlerwill break as soon as Bundler drops support for your Ruby version. Please upgrade RubyGems to avoid future breakage and silence this warning by runninggem update --system 3.2.3A server is already running. Check /usr/src/app/tmp/pids/server.pid. Your RubyGems version (3.0.3.1) has a bug that preventsrequired_ruby_versionfrom working for Bundler. Any scripts that usegem install bundlerwill break as soon as Bundler drops support for your Ruby version. Please upgrade RubyGems to avoid future breakage and silence this warning by runninggem update --system 3.2.3 A server is already running. Check /usr/src/app/tmp/pids/server.pid.

Depuis le logfile error.log, je retrouve notamment des erreurs « acces denied » à destination de l’instance docker fab-manager sur le port 3000.

2023/03/07 23:37:06 [error] 23#23: *64 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.56.10, server: fablab.fablimagne.fr, request: "GET /favicon.ico HTTP/1.1", upstream: "http://172.18.0.6:3000/favicon.ico", host: "fablab.fablimagne.fr", referrer: "https://fablab.fablimagne.fr/" 2023/03/07 23:37:45 [error] 23#23: *67 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.56.10, server: fablab.fablimagne.fr, request: "GET / HTTP/1.1", upstream: "http://172.18.0.6:3000/", host: "fablab.fablimagne.fr", referrer: "https://www.plainelimagne.com/" 2023/03/07 23:37:45 [error] 23#23: *67 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.56.10, server: fablab.fablimagne.fr, request: "GET /favicon.ico HTTP/1.1", upstream: "http://172.18.0.6:3000/favicon.ico", host: "fablab.fablimagne.fr", referrer: "https://fablab.fablimagne.fr/" 2023/03/07 23:38:42 [error] 23#23: *70 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.56.10, server: fablab.fablimagne.fr, request: "GET / HTTP/1.1", upstream: "http://172.18.0.6:3000/", host: "fablab.fablimagne.fr", referrer: "https://www.plainelimagne.com/" 2023/03/07 23:38:42 [error] 23#23: *70 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.56.10, server: fablab.fablimagne.fr, request: "GET /favicon.ico HTTP/1.1", upstream: "http://172.18.0.6:3000/favicon.ico", host: "fablab.fablimagne.fr", referrer: "https://fablab.fablimagne.fr/" 2023/03/07 23:40:45 [error] 23#23: *73 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.56.10, server: fablab.fablimagne.fr, request: "GET / HTTP/1.1", upstream: "http://172.18.0.6:3000/", host: "fablab.fablimagne.fr", referrer: "https://www.plainelimagne.com/" 2023/03/07 23:40:45 [error] 23#23: *73 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.56.10, server: fablab.fablimagne.fr, request: "GET /favicon.ico HTTP/1.1", upstream: "http://172.18.0.6:3000/favicon.ico", host: "fablab.fablimagne.fr", referrer: "https://fablab.fablimagne.fr/"

Après un redémarrage des containers via docker-compose down et docker-compose up -d, tout reviens à la normal (j’avais même cron cela tous les soirs à 2H00, mais les plantages sont à des heures aléatoires visiblement). Au redémarrage, je constate les mentions suivantes dans le logfile app-stderr.log

bundler: failed to load command: rails (/usr/local/bundle/bin/rails) /usr/local/bundle/gems/bundler-2.4.7/lib/bundler/definition.rb:524:in materialize’: Could not find bootsnap-1.4.6, puma-4.3.12, json-2.3.1, pg-1.2.2, oj-3.10.5, sha3-1.0.1, msgpack-1.3.3, nio4r-2.5.8, bcrypt-3.1.13, mimemagic-0.4.3, ffi-1.15.5, websocket-driver-0.7.5, racc-1.6.1 in locally installed gems (Bundler::GemNotFound)
from /usr/local/bundle/gems/bundler-2.4.7/lib/bundler/definition.rb:197:in specs' from /usr/local/bundle/gems/bundler-2.4.7/lib/bundler/definition.rb:254:in specs_for’
from /usr/local/bundle/gems/bundler-2.4.7/lib/bundler/runtime.rb:18:in setup' from /usr/local/bundle/gems/bundler-2.4.7/lib/bundler.rb:170:in setup’
from /usr/local/bundle/gems/bundler-2.4.7/lib/bundler/setup.rb:20:in block in <top (required)>' from /usr/local/bundle/gems/bundler-2.4.7/lib/bundler/ui/shell.rb:159:in with_level’
from /usr/local/bundle/gems/bundler-2.4.7/lib/bundler/ui/shell.rb:111:in silence' from /usr/local/bundle/gems/bundler-2.4.7/lib/bundler/setup.rb:20:in <top (required)>’
from /usr/local/bundle/gems/bundler-2.4.7/lib/bundler/cli/exec.rb:56:in require_relative' from /usr/local/bundle/gems/bundler-2.4.7/lib/bundler/cli/exec.rb:56:in kernel_load’
from /usr/local/bundle/gems/bundler-2.4.7/lib/bundler/cli/exec.rb:23:in run' from /usr/local/bundle/gems/bundler-2.4.7/lib/bundler/cli.rb:492:in exec’
from /usr/local/bundle/gems/bundler-2.4.7/lib/bundler/vendor/thor/lib/thor/command.rb:27:in run' from /usr/local/bundle/gems/bundler-2.4.7/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in invoke_command’
from /usr/local/bundle/gems/bundler-2.4.7/lib/bundler/vendor/thor/lib/thor.rb:392:in dispatch' from /usr/local/bundle/gems/bundler-2.4.7/lib/bundler/cli.rb:34:in dispatch’
from /usr/local/bundle/gems/bundler-2.4.7/lib/bundler/vendor/thor/lib/thor/base.rb:485:in start' from /usr/local/bundle/gems/bundler-2.4.7/lib/bundler/cli.rb:28:in start’
from /usr/local/bundle/gems/bundler-2.4.7/exe/bundle:45:in block in <top (required)>' from /usr/local/bundle/gems/bundler-2.4.7/lib/bundler/friendly_errors.rb:117:in with_friendly_errors’
from /usr/local/bundle/gems/bundler-2.4.7/exe/bundle:33:in <top (required)>' from /usr/local/bundle/bin/bundle:23:in load’
from /usr/local/bundle/bin/bundle:23:in <main>' bundler: failed to load command: rails (/usr/local/bundle/bin/rails) /usr/local/bundle/gems/bundler-2.4.7/lib/bundler/definition.rb:524:in materialize’: Could not find bootsnap-1.4.6, puma-4.3.12, json-2.3.1, pg-1.2.2, oj-3.10.5, sha3-1.0.1, msgpack-1.3.3, nio4r-2.5.8, bcrypt-3.1.13, mimemagic-0.4.3, ffi-1.15.5, websocket-driver-0.7.5, racc-1.6.1 in locally installed gems (Bundler::GemNotFound)
from /usr/local/bundle/gems/bundler-2.4.7/lib/bundler/definition.rb:197:in specs' from /usr/local/bundle/gems/bundler-2.4.7/lib/bundler/definition.rb:254:in specs_for’
from /usr/local/bundle/gems/bundler-2.4.7/lib/bundler/runtime.rb:18:in setup' from /usr/local/bundle/gems/bundler-2.4.7/lib/bundler.rb:170:in setup’
from /usr/local/bundle/gems/bundler-2.4.7/lib/bundler/setup.rb:20:in block in <top (required)>' from /usr/local/bundle/gems/bundler-2.4.7/lib/bundler/ui/shell.rb:159:in with_level’
from /usr/local/bundle/gems/bundler-2.4.7/lib/bundler/ui/shell.rb:111:in silence' from /usr/local/bundle/gems/bundler-2.4.7/lib/bundler/setup.rb:20:in <top (required)>’
from /usr/local/bundle/gems/bundler-2.4.7/lib/bundler/cli/exec.rb:56:in require_relative' from /usr/local/bundle/gems/bundler-2.4.7/lib/bundler/cli/exec.rb:56:in kernel_load’
from /usr/local/bundle/gems/bundler-2.4.7/lib/bundler/cli/exec.rb:23:in run' from /usr/local/bundle/gems/bundler-2.4.7/lib/bundler/cli.rb:492:in exec’
from /usr/local/bundle/gems/bundler-2.4.7/lib/bundler/vendor/thor/lib/thor/command.rb:27:in run' from /usr/local/bundle/gems/bundler-2.4.7/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in invoke_command’
from /usr/local/bundle/gems/bundler-2.4.7/lib/bundler/vendor/thor/lib/thor.rb:392:in dispatch' from /usr/local/bundle/gems/bundler-2.4.7/lib/bundler/cli.rb:34:in dispatch’
from /usr/local/bundle/gems/bundler-2.4.7/lib/bundler/vendor/thor/lib/thor/base.rb:485:in start' from /usr/local/bundle/gems/bundler-2.4.7/lib/bundler/cli.rb:28:in start’
from /usr/local/bundle/gems/bundler-2.4.7/exe/bundle:45:in block in <top (required)>' from /usr/local/bundle/gems/bundler-2.4.7/lib/bundler/friendly_errors.rb:117:in with_friendly_errors’
from /usr/local/bundle/gems/bundler-2.4.7/exe/bundle:33:in <top (required)>' from /usr/local/bundle/bin/bundle:23:in load’
from /usr/local/bundle/bin/bundle:23:in <main>' bundler: failed to load command: rails (/usr/local/bundle/bin/rails) /usr/local/bundle/gems/bundler-2.4.7/lib/bundler/definition.rb:524:in materialize’: Could not find bootsnap-1.4.6, puma-4.3.12, json-2.3.1, pg-1.2.2, oj-3.10.5, sha3-1.0.1, msgpack-1.3.3, nio4r-2.5.8, bcrypt-3.1.13, mimemagic-0.4.3, ffi-1.15.5, websocket-driver-0.7.5, racc-1.6.1 in locally installed gems (Bundler::GemNotFound)
from /usr/local/bundle/gems/bundler-2.4.7/lib/bundler/definition.rb:197:in specs' from /usr/local/bundle/gems/bundler-2.4.7/lib/bundler/definition.rb:254:in specs_for’
from /usr/local/bundle/gems/bundler-2.4.7/lib/bundler/runtime.rb:18:in setup' from /usr/local/bundle/gems/bundler-2.4.7/lib/bundler.rb:170:in setup’
from /usr/local/bundle/gems/bundler-2.4.7/lib/bundler/setup.rb:20:in block in <top (required)>' from /usr/local/bundle/gems/bundler-2.4.7/lib/bundler/ui/shell.rb:159:in with_level’
from /usr/local/bundle/gems/bundler-2.4.7/lib/bundler/ui/shell.rb:111:in silence' from /usr/local/bundle/gems/bundler-2.4.7/lib/bundler/setup.rb:20:in <top (required)>’
from /usr/local/bundle/gems/bundler-2.4.7/lib/bundler/cli/exec.rb:56:in require_relative' from /usr/local/bundle/gems/bundler-2.4.7/lib/bundler/cli/exec.rb:56:in kernel_load’
from /usr/local/bundle/gems/bundler-2.4.7/lib/bundler/cli/exec.rb:23:in run' from /usr/local/bundle/gems/bundler-2.4.7/lib/bundler/cli.rb:492:in exec’
from /usr/local/bundle/gems/bundler-2.4.7/lib/bundler/vendor/thor/lib/thor/command.rb:27:in run' from /usr/local/bundle/gems/bundler-2.4.7/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in invoke_command’
from /usr/local/bundle/gems/bundler-2.4.7/lib/bundler/vendor/thor/lib/thor.rb:392:in dispatch' from /usr/local/bundle/gems/bundler-2.4.7/lib/bundler/cli.rb:34:in dispatch’
from /usr/local/bundle/gems/bundler-2.4.7/lib/bundler/vendor/thor/lib/thor/base.rb:485:in start' from /usr/local/bundle/gems/bundler-2.4.7/lib/bundler/cli.rb:28:in start’
from /usr/local/bundle/gems/bundler-2.4.7/exe/bundle:45:in block in <top (required)>' from /usr/local/bundle/gems/bundler-2.4.7/lib/bundler/friendly_errors.rb:117:in with_friendly_errors’
from /usr/local/bundle/gems/bundler-2.4.7/exe/bundle:33:in <top (required)>' from /usr/local/bundle/bin/bundle:23:in load’
from /usr/local/bundle/bin/bundle:23:in <main>' bundler: failed to load command: rails (/usr/local/bundle/bin/rails) /usr/local/bundle/gems/bundler-2.4.7/lib/bundler/definition.rb:524:in materialize’: Could not find bootsnap-1.4.6, puma-4.3.12, json-2.3.1, pg-1.2.2, oj-3.10.5, sha3-1.0.1, msgpack-1.3.3, nio4r-2.5.8, bcrypt-3.1.13, mimemagic-0.4.3, ffi-1.15.5, websocket-driver-0.7.5, racc-1.6.1 in locally installed gems (Bundler::GemNotFound)
from /usr/local/bundle/gems/bundler-2.4.7/lib/bundler/definition.rb:197:in specs' from /usr/local/bundle/gems/bundler-2.4.7/lib/bundler/definition.rb:254:in specs_for’
from /usr/local/bundle/gems/bundler-2.4.7/lib/bundler/runtime.rb:18:in setup' from /usr/local/bundle/gems/bundler-2.4.7/lib/bundler.rb:170:in setup’
from /usr/local/bundle/gems/bundler-2.4.7/lib/bundler/setup.rb:20:in block in <top (required)>' from /usr/local/bundle/gems/bundler-2.4.7/lib/bundler/ui/shell.rb:159:in with_level’
from /usr/local/bundle/gems/bundler-2.4.7/lib/bundler/ui/shell.rb:111:in silence' from /usr/local/bundle/gems/bundler-2.4.7/lib/bundler/setup.rb:20:in <top (required)>’
from /usr/local/bundle/gems/bundler-2.4.7/lib/bundler/cli/exec.rb:56:in require_relative' from /usr/local/bundle/gems/bundler-2.4.7/lib/bundler/cli/exec.rb:56:in kernel_load’
from /usr/local/bundle/gems/bundler-2.4.7/lib/bundler/cli/exec.rb:23:in run' from /usr/local/bundle/gems/bundler-2.4.7/lib/bundler/cli.rb:492:in exec’
from /usr/local/bundle/gems/bundler-2.4.7/lib/bundler/vendor/thor/lib/thor/command.rb:27:in run' from /usr/local/bundle/gems/bundler-2.4.7/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in invoke_command’
from /usr/local/bundle/gems/bundler-2.4.7/lib/bundler/vendor/thor/lib/thor.rb:392:in dispatch' from /usr/local/bundle/gems/bundler-2.4.7/lib/bundler/cli.rb:34:in dispatch’
from /usr/local/bundle/gems/bundler-2.4.7/lib/bundler/vendor/thor/lib/thor/base.rb:485:in start' from /usr/local/bundle/gems/bundler-2.4.7/lib/bundler/cli.rb:28:in start’
from /usr/local/bundle/gems/bundler-2.4.7/exe/bundle:45:in block in <top (required)>' from /usr/local/bundle/gems/bundler-2.4.7/lib/bundler/friendly_errors.rb:117:in with_friendly_errors’
from /usr/local/bundle/gems/bundler-2.4.7/exe/bundle:33:in <top (required)>' from /usr/local/bundle/bin/bundle:23:in load’
from /usr/local/bundle/bin/bundle:23:in <main>'

J’étais à jour jusqu’à la version 5.7.2 et viens tout juste de passer en 5.8.0 dans l’espoir d’obtenir quelques choses qui me corrige ces erreurs, mais j’admets ne pas vraiment y croire (j’espérais déjà cela en passant sur 5.7.2 assez récemment).

J’ai bien essayé l’exécution de la commande gems dans le shell du container de fab-manager, mais cela ne semble pas mieux fonctionner …

Je m’en remets à vous car j’exploite peu Docker, et ABSOLUMENT pas Rails et Elastic-Search … Je suis tristement bloqué à ne pas comprendre le pourquoi du comment malgré être sur un environnement Linux qui m’est très familier (les containers me sont encore un peu étranger, je suis fidèle au KVM ^^')

Nouveau plantage et rien de nouveau dans les logs … Je comprends difficilement ce qu’il se passe.

Salut, tu as combien de mémoire vive sur ta machine ?

Bonjour Sylvain,

L’environnement tourne dans une machine virtuelle KVM sur Proxmox est dispose de 4 Go de RAM et 2 coeurs. Actuellement, la machine virtuelle semble ne pas dépasser les 1,4 Go d’utilisation (buffer compris), mais il est possible que les instances fassent des pics de consommation (peut-être ?)

Techniquement, j’ai encore assez de ressources a allouer si ce n’est pas suffisant.

J’ajouterai juste ne pas avoir alloué de swap au système (je préfère rajouter de la RAM que d’utiliser les SSD qui sont en 3 réplications via du Bluestore CEPH, cela me ronge les IO de manière aberrante …)

Hier soir, mise à jour vers 5.9.0, augmentation des cores de 2 à 4, augmentation de la RAM de 4 Go à 8 Go :

  • 2h00 : tout fonctionne avec une détection de la version 5.9.0
  • 9h30 : énième et même plantage, toujours les mêmes logs …

Salut @AZiiDok,

C’est très bizarre… peux-tu quand même tenter une mise à jour à Fab-manager v6 qui utilise version plus récente de ruby et du serveur puma. Peut-être que cela va résoudre ton problème ?

Bonjour @Sylvain

J’ai finalement « l’impression » d’avoir résolu mon problème en résolvant un autre : lorsque l’on se rendait dans les notifications, il y avait un message ‹ Internal Server Error ›.

« Un simple select qui fait une erreur ? Ca ressemble à un problème d’enregistrement … Je purge la table » et depuis, il n’a pas replanté.

Lien de cause à effet : je ne sais pas, mais je n’ai pas eu besoin de le redémarrer depuis maintenant presque 1 semaine, et les notifications refonctionnent.

Je reviendrais sur ce post dans quelques jours/semaines pour le clôturer et conclure de manière définitive si la solution a finalement fonctionnée. Ca m’a paru étrange que cela puisse venir de la table Postgres des notifications, mais si un enregistrement était mauvais et qu’un script de mailing s’exécutait, ça peut devenir cohérent.

En effet c’est assez étrange … peut-être un enregistrement de base de données corrompu à cause d’un problème de file system, j’ai déjà vu ça, mais ça ne faisait pas redémarrer l’app ! En tout cas cool si ça refonctionne, et merci pour ton partage d’expérience