This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revision | ||
production:shaman.ceph.com [2019/10/09 15:23] djgalloway |
production:shaman.ceph.com [2023/04/06 00:51] (current) dmick [Summary] |
||
---|---|---|---|
Line 4: | Line 4: | ||
There are three VMs in the [[https://wiki.sepia.ceph.com/doku.php?id=services:ovh#production_services|OVH CI region]] that make up shaman. | There are three VMs in the [[https://wiki.sepia.ceph.com/doku.php?id=services:ovh#production_services|OVH CI region]] that make up shaman. | ||
- | * shaman.ceph.com is just a load balancing VM | + | * shaman.ceph.com is just a load balancing VM. Accesses are proxied to either 1.shaman.ceph.com or 2.shaman.ceph.com with an 'upstream shaman' clause in /etc/nginx/nginx.conf, which is then referred to by the site config for shaman.ceph.com. |
* 1.shaman.ceph.com is the primary shaman node that has the postgres DB with all the repo information | * 1.shaman.ceph.com is the primary shaman node that has the postgres DB with all the repo information | ||
* 2.shaman.ceph.com is a **READ ONLY** backup in the event 1.shaman.ceph.com goes down | * 2.shaman.ceph.com is a **READ ONLY** backup in the event 1.shaman.ceph.com goes down | ||
+ | * 2.shaman.ceph.com can handle write requests because pecan, the web framework, is also aware of the primary/hot standby configuration, and so will redirect writes to 1.shaman.ceph.com on its own, if they appear. | ||
===== User Access ===== | ===== User Access ===== | ||
Line 43: | Line 44: | ||
==== Pulling Slave Stats ==== | ==== Pulling Slave Stats ==== | ||
I needed to determine what percentage of jobs were running on static vs. ephemeral slaves. Alfredo wrote a python script to pull this data out of the shaman database. This script totals how many jobs ran on static vs. ephemeral slaves over a 2 week period (since that's how long we keep dev builds). | I needed to determine what percentage of jobs were running on static vs. ephemeral slaves. Alfredo wrote a python script to pull this data out of the shaman database. This script totals how many jobs ran on static vs. ephemeral slaves over a 2 week period (since that's how long we keep dev builds). | ||
+ | |||
+ | Doing this on 2.shaman.ceph.com ensures you're in a read-only capacity. | ||
- ''ssh 2.shaman.ceph.com'' | - ''ssh 2.shaman.ceph.com'' | ||
Line 62: | Line 65: | ||
ovh_builds = {} | ovh_builds = {} | ||
- | lab_builds = {} | + | irvingi_builds = {} |
+ | braggi_builds = {} | ||
+ | adami_builds = {} | ||
rest_of_the_world = {} | rest_of_the_world = {} | ||
ovh_count = 0 | ovh_count = 0 | ||
- | lab_count = 0 | + | irvingi_count = 0 |
+ | braggi_count = 0 | ||
+ | adami_count = 0 | ||
for build in builds: | for build in builds: | ||
Line 73: | Line 80: | ||
counter = ovh_count | counter = ovh_count | ||
elif 'slave-' in node_name: | elif 'slave-' in node_name: | ||
- | mapping = lab_builds | + | mapping = irvingi_builds |
- | counter = lab_count | + | counter = irvingi_count |
+ | elif 'braggi' in node_name: | ||
+ | mapping = braggi_builds | ||
+ | counter = braggi_count | ||
+ | elif 'adami' in node_name: | ||
+ | mapping = adami_builds | ||
+ | counter = adami_count | ||
else: | else: | ||
mapping = rest_of_the_world | mapping = rest_of_the_world | ||
Line 83: | Line 96: | ||
mapping[node_name] = 1 | mapping[node_name] = 1 | ||
| | ||
- | for mapping in [ovh_builds, lab_builds]: | + | for mapping in [ovh_builds, irvingi_builds, braggi_builds, adami_builds]: |
count = 0 | count = 0 | ||
for key, value in mapping.items(): | for key, value in mapping.items(): | ||
Line 92: | Line 105: | ||
</code> | </code> | ||
- | - ''%%pecan shell --shell ipython prod.py%%'' | + | - ''%%/opt/shaman/bin/pecan shell --shell ipython prod.py%%'' |
- Then<code> | - Then<code> | ||
In [1]: from shaman import models | In [1]: from shaman import models | ||
Line 101: | Line 114: | ||
In [4]: two_week_stats.report() | In [4]: two_week_stats.report() | ||
+ | </code> | ||
+ | |||
+ | ==== Delete builds/repos from database ==== | ||
+ | HYPOTHETICALLY ;-) if a repo/build got pushed to shaman that contains an embargoed security fix, you can delete the entries from shaman's DB. The packages will still be on chacra servers but shaman won't know about them. You can always [[production:chacra.ceph.com#manually_delete_a_repo_from_postgres_db|delete]] them from chacra too if necessary. | ||
+ | |||
+ | <code> | ||
+ | ssh 1.shaman.ceph.com | ||
+ | sudo su - postgres | ||
+ | |||
+ | postgres@1:~$ psql -d shaman | ||
+ | psql (9.5.23) | ||
+ | Type "help" for help. | ||
+ | |||
+ | shaman=# \dt | ||
+ | List of relations | ||
+ | Schema | Name | Type | Owner | ||
+ | --------+-----------------+-------+-------- | ||
+ | public | alembic_version | table | shaman | ||
+ | public | archs | table | shaman | ||
+ | public | builds | table | shaman | ||
+ | public | nodes | table | shaman | ||
+ | public | projects | table | shaman | ||
+ | public | repos | table | shaman | ||
+ | (6 rows) | ||
+ | |||
+ | shaman=# delete from public.builds where sha1 = 'f73b19678311b996984c30e7c0eb96a22ffa29ce'; | ||
+ | DELETE 6 | ||
+ | |||
+ | shaman=# select id from public.repos where sha1 = 'f73b19678311b996984c30e7c0eb96a22ffa29ce'; | ||
+ | id | ||
+ | -------- | ||
+ | 197001 | ||
+ | 197010 | ||
+ | 197011 | ||
+ | 197012 | ||
+ | 197030 | ||
+ | 196999 | ||
+ | |||
+ | shaman=# delete from public.archs where repo_id = '197001'; | ||
+ | DELETE 1 | ||
+ | shaman=# delete from public.archs where repo_id = '197010'; | ||
+ | DELETE 2 | ||
+ | shaman=# delete from public.archs where repo_id = '197011'; | ||
+ | DELETE 2 | ||
+ | shaman=# delete from public.archs where repo_id = '197012'; | ||
+ | DELETE 2 | ||
+ | shaman=# delete from public.archs where repo_id = '197030'; | ||
+ | DELETE 2 | ||
+ | shaman=# delete from public.archs where repo_id = '196999'; | ||
+ | DELETE 1 | ||
+ | shaman=# delete from public.repos where sha1 = 'f73b19678311b996984c30e7c0eb96a22ffa29ce'; | ||
+ | DELETE 6 | ||
</code> | </code> | ||