Today, after shutting down Postgres, to add memory to the machine, it refused to start.
Notes like:
1Mar 2 03:50:31 pgsql postgres[1234]: [3-1]: FATAL: the database system is starting up
scrolled by quickly and no apparent progress was made. Eventually, cntl-c stopped things. Inspecting the log showed nothing (literally, empty log file)
Single user mode
After a bit of googling (mostly useless, no logs to narrow down anything), I went back to reading the man page for postgres. It offered a single user mode of operation (similar in concept to the single user mode that UNIX offers). I ended up using the following options:
–single: single user mode
-D: path to the data directory
-P: disable system indexes
-d 1: set debug output to 1
1$ /usr/local/bin/postgres --single -D /usr/local/pgsql/data -P -d 1
2LOG: database system was shut down in recovery at 2017-03-02 03:53:05 UTC
3DEBUG: checkpoint record is at 1A5/EEA5ED0
4DEBUG: redo record is at 1A4/FB0DC7F8; shutdown FALSE
5DEBUG: next transaction ID: 0:69624573; next OID: 191021
6DEBUG: next MultiXactId: 17; next MultiXactOffset: 33
7DEBUG: oldest unfrozen transaction ID: 853, in database 12716
8DEBUG: oldest MultiXactId: 1, in database 27036
9DEBUG: commit timestamp Xid oldest/newest: 0/0
10DEBUG: transaction ID wrap limit is 2147484500, limited by database with OID 12716
11DEBUG: MultiXactId wrap limit is 2147483648, limited by database with OID 27036
12DEBUG: starting up replication slots
13LOG: database system was not properly shut down; automatic recovery in progress
14DEBUG: resetting unlogged relations: cleanup 1 init 0
15LOG: redo starts at 1A4/FB0DC7F8
16LOG: invalid record length at 1A5/23DF9F30: wanted 24, got 0
17LOG: redo done at 1A5/23DF9F08
18LOG: last completed transaction was at log time 2017-03-02 03:26:45.090211+00
19DEBUG: resetting unlogged relations: cleanup 0 init 1
20LOG: checkpoint starting: end-of-recovery immediate
21DEBUG: performing replication slot checkpoint
22DEBUG: checkpoint sync: number=1 file=base/16762/77539 time=0.030 msec
23DEBUG: checkpoint sync: number=2 file=base/16762/77902 time=0.024 msec
24DEBUG: checkpoint sync: number=3 file=base/16762/77569 time=0.020 msec
25DEBUG: checkpoint sync: number=4 file=base/16762/77877 time=0.021 msec
26DEBUG: checkpoint sync: number=5 file=base/82382/71522 time=0.024 msec
27DEBUG: checkpoint sync: number=6 file=base/82382/71522_fsm time=0.047 msec
28DEBUG: checkpoint sync: number=7 file=base/16385/69196 time=0.039 msec
29DEBUG: checkpoint sync: number=8 file=base/16762/77878 time=0.022 msec
30DEBUG: checkpoint sync: number=9 file=base/82382/82468 time=0.020 msec
31DEBUG: checkpoint sync: number=10 file=base/133179/71525 time=0.023 msec
32DEBUG: checkpoint sync: number=11 file=base/133179/71525_fsm time=0.022 msec
33DEBUG: checkpoint sync: number=12 file=base/133179/71525_vm time=0.020 msec
34DEBUG: checkpoint sync: number=12 file=base/133179/71525_vm time=0.020 msec
35DEBUG: checkpoint sync: number=13 file=base/82382/83786 time=0.006 msec
36DEBUG: checkpoint sync: number=14 file=base/16762/77444 time=0.020 msec
37DEBUG: checkpoint sync: number=15 file=base/16762/77370 time=0.019 msec
38DEBUG: checkpoint sync: number=16 file=base/16762/77370_vm time=0.019 msec
39DEBUG: checkpoint sync: number=17 file=base/16762/77527 time=0.019 msec
40DEBUG: checkpoint sync: number=18 file=base/16762/77527_fsm time=0.020 msec
41DEBUG: checkpoint sync: number=19 file=base/16762/77527_vm time=0.020 msec
42DEBUG: checkpoint sync: number=20 file=base/82382/83536 time=0.020 msec
43DEBUG: checkpoint sync: number=21 file=base/16762/77873 time=0.020 msec
44DEBUG: checkpoint sync: number=22 file=base/16762/77420 time=0.019 msec
45DEBUG: checkpoint sync: number=23 file=base/16762/77420_fsm time=0.019 msec
46DEBUG: checkpoint sync: number=24 file=base/16762/77420_vm time=0.019 msec
47DEBUG: checkpoint sync: number=25 file=base/27036/68257 time=0.020 msec
48DEBUG: checkpoint sync: number=26 file=base/27036/68257_fsm time=0.019 msec
49DEBUG: checkpoint sync: number=27 file=base/27036/68257_vm time=0.019 msec
50DEBUG: checkpoint sync: number=28 file=base/27036/68269 time=0.019 msec
51DEBUG: checkpoint sync: number=29 file=base/27036/68269_fsm time=0.018 msec
52DEBUG: checkpoint sync: number=30 file=base/27036/68269_vm time=0.018 msec
53DEBUG: checkpoint sync: number=31 file=base/82382/83458 time=0.020 msec
54DEBUG: checkpoint sync: number=32 file=base/16762/69411 time=0.019 msec
55DEBUG: checkpoint sync: number=33 file=base/16762/69411_fsm time=0.018 msec
56DEBUG: checkpoint sync: number=34 file=base/16762/69411_vm time=0.018 msec
57DEBUG: checkpoint sync: number=35 file=base/82382/83803 time=0.018 msec
58DEBUG: checkpoint sync: number=36 file=base/82382/83803_fsm time=0.019 msec
59DEBUG: checkpoint sync: number=37 file=base/16762/77566 time=0.019 msec
60DEBUG: checkpoint sync: number=38 file=base/16762/77471 time=0.020 msec
61DEBUG: checkpoint sync: number=39 file=base/16762/77471_vm time=0.019 msec
62DEBUG: checkpoint sync: number=40 file=base/16762/77875 time=0.019 msec
63DEBUG: checkpoint sync: number=41 file=base/82382/82465 time=0.018 msec
64DEBUG: checkpoint sync: number=42 file=base/16762/77860 time=0.019 msec
65DEBUG: checkpoint sync: number=43 file=base/16762/77860_fsm time=0.019 msec
66DEBUG: checkpoint sync: number=44 file=base/16762/77860_vm time=0.019 msec
67DEBUG: checkpoint sync: number=45 file=base/16762/77418 time=0.018 msec
68DEBUG: checkpoint sync: number=46 file=base/82382/83482 time=0.019 msec
69DEBUG: checkpoint sync: number=47 file=base/82382/83823 time=0.018 msec
70DEBUG: checkpoint sync: number=48 file=base/16762/69547 time=0.018 msec
71DEBUG: checkpoint sync: number=49 file=base/16762/77447 time=0.020 msec
72DEBUG: checkpoint sync: number=50 file=base/82382/83558 time=0.018 msec
73DEBUG: checkpoint sync: number=51 file=base/27036/186065 time=289.835 msec
74DEBUG: checkpoint sync: number=52 file=base/27036/68623 time=0.026 msec
75DEBUG: checkpoint sync: number=53 file=base/16762/77541 time=0.020 msec
76DEBUG: checkpoint sync: number=54 file=base/82382/82467 time=0.022 msec
77DEBUG: checkpoint sync: number=55 file=base/82382/82463 time=0.019 msec
78DEBUG: checkpoint sync: number=56 file=base/16762/77876 time=0.019 msec
79DEBUG: checkpoint sync: number=57 file=base/82382/82689 time=0.020 msec
80DEBUG: checkpoint sync: number=58 file=base/82382/82689_fsm time=0.019 msec
81DEBUG: checkpoint sync: number=59 file=base/82382/83901 time=0.018 msec
82DEBUG: checkpoint sync: number=60 file=base/133179/71522 time=0.019 msec
83DEBUG: checkpoint sync: number=61 file=base/133179/71522_fsm time=0.020 msec
84DEBUG: checkpoint sync: number=62 file=base/133179/71522_vm time=0.018 msec
85DEBUG: checkpoint sync: number=63 file=base/16762/77903 time=0.018 msec
86DEBUG: checkpoint sync: number=64 file=base/16762/77635 time=0.019 msec
87DEBUG: checkpoint sync: number=65 file=base/16762/77635_fsm time=0.018 msec
88DEBUG: checkpoint sync: number=66 file=base/16762/77635_vm time=0.019 msec
89DEBUG: checkpoint sync: number=67 file=base/82382/82590 time=0.019 msec
90DEBUG: checkpoint sync: number=68 file=base/82382/83438 time=0.019 msec
91DEBUG: checkpoint sync: number=69 file=base/82382/83438_fsm time=0.019 msec
92DEBUG: checkpoint sync: number=70 file=base/82382/83438_vm time=0.019 msec
93DEBUG: checkpoint sync: number=71 file=base/82382/83538 time=0.018 msec
94DEBUG: checkpoint sync: number=72 file=base/82382/83789 time=0.018 msec
95DEBUG: checkpoint sync: number=73 file=base/82382/83825 time=0.018 msec
96DEBUG: checkpoint sync: number=74 file=base/16762/77446 time=0.019 msec
97DEBUG: checkpoint sync: number=75 file=base/82382/83452 time=0.018 msec
98DEBUG: checkpoint sync: number=76 file=base/82382/83452_fsm time=0.018 msec
99DEBUG: checkpoint sync: number=77 file=base/16762/77880 time=0.025 msec
100DEBUG: checkpoint sync: number=78 file=base/16762/77880_fsm time=0.018 msec
101DEBUG: checkpoint sync: number=79 file=base/16762/77880_vm time=0.020 msec
102DEBUG: checkpoint sync: number=80 file=base/16385/68648 time=0.019 msec
103DEBUG: checkpoint sync: number=81 file=base/16385/68648_fsm time=0.019 msec
104DEBUG: checkpoint sync: number=82 file=base/16385/68648_vm time=0.019 msec
105DEBUG: checkpoint sync: number=83 file=base/82382/83667 time=0.018 msec
106DEBUG: checkpoint sync: number=84 file=base/27036/68247 time=0.018 msec
107DEBUG: checkpoint sync: number=85 file=base/27036/68247_vm time=0.019 msec
108DEBUG: checkpoint sync: number=86 file=base/16762/69408 time=0.019 msec
109DEBUG: checkpoint sync: number=87 file=base/16762/69408_fsm time=0.019 msec
110DEBUG: checkpoint sync: number=88 file=base/16762/69408_vm time=0.018 msec
111DEBUG: checkpoint sync: number=89 file=base/82382/83526 time=0.019 msec
112DEBUG: checkpoint sync: number=90 file=base/82382/83526_fsm time=0.018 msec
113DEBUG: checkpoint sync: number=91 file=base/82382/83548 time=0.018 msec
114DEBUG: checkpoint sync: number=92 file=base/82382/83669 time=0.019 msec
115DEBUG: checkpoint sync: number=93 file=base/16762/77568 time=0.018 msec
116DEBUG: checkpoint sync: number=94 file=base/82382/82466 time=0.019 msec
117DEBUG: checkpoint sync: number=95 file=base/16762/77552 time=0.019 msec
118DEBUG: checkpoint sync: number=96 file=base/16762/77552_fsm time=0.018 msec
119DEBUG: checkpoint sync: number=97 file=base/16762/77552_vm time=0.018 msec
120DEBUG: checkpoint sync: number=98 file=base/16762/77633 time=0.018 msec
121DEBUG: checkpoint sync: number=99 file=base/27036/68262 time=0.019 msec
122DEBUG: checkpoint sync: number=100 file=base/16762/77641 time=0.018 msec
123DEBUG: checkpoint sync: number=101 file=base/16762/77899 time=0.019 msec
124DEBUG: checkpoint sync: number=102 file=base/82382/83459 time=0.019 msec
125DEBUG: checkpoint sync: number=103 file=base/27036/68263 time=0.019 msec
126DEBUG: checkpoint sync: number=104 file=base/16762/77550 time=0.019 msec
127DEBUG: checkpoint sync: number=104 file=base/16762/77550 time=0.019 msec
128DEBUG: checkpoint sync: number=105 file=base/82382/83472 time=0.019 msec
129DEBUG: checkpoint sync: number=106 file=base/82382/83788 time=0.018 msec
130DEBUG: checkpoint sync: number=107 file=base/82382/71528 time=0.020 msec
131DEBUG: checkpoint sync: number=108 file=base/133179/71646 time=0.018 msec
132DEBUG: checkpoint sync: number=109 file=base/16385/68645 time=0.018 msec
133DEBUG: checkpoint sync: number=110 file=base/16385/68645_fsm time=0.026 msec
134DEBUG: checkpoint sync: number=111 file=base/16385/68645_vm time=0.019 msec
135DEBUG: checkpoint sync: number=112 file=base/133179/71527 time=0.018 msec
136DEBUG: checkpoint sync: number=113 file=base/82382/82575 time=0.019 msec
137DEBUG: checkpoint sync: number=114 file=base/16762/77901 time=0.018 msec
138DEBUG: checkpoint sync: number=115 file=base/27036/68625 time=0.020 msec
139DEBUG: checkpoint sync: number=116 file=base/133179/71528 time=0.020 msec
140DEBUG: checkpoint sync: number=117 file=base/27036/186066 time=0.907 msec
141DEBUG: checkpoint sync: number=118 file=base/82382/83445.1 time=0.022 msec
142DEBUG: checkpoint sync: number=119 file=base/82382/83445.2 time=0.019 msec
143DEBUG: checkpoint sync: number=120 file=base/82382/83445_fsm time=0.021 msec
144DEBUG: checkpoint sync: number=121 file=base/16385/68650 time=0.021 msec
145DEBUG: checkpoint sync: number=122 file=base/82382/83473 time=0.018 msec
146DEBUG: checkpoint sync: number=123 file=base/82382/83473_fsm time=0.109 msec
147DEBUG: checkpoint sync: number=124 file=base/27036/68624 time=0.019 msec
148DEBUG: checkpoint sync: number=125 file=base/82382/71646 time=0.021 msec
149DEBUG: checkpoint sync: number=126 file=base/82382/83682 time=0.018 msec
150DEBUG: checkpoint sync: number=127 file=base/16762/77858 time=0.019 msec
151DEBUG: checkpoint sync: number=128 file=base/27036/68620 time=0.019 msec
152DEBUG: checkpoint sync: number=129 file=base/27036/68620_vm time=0.019 msec
153DEBUG: checkpoint sync: number=130 file=base/16385/68657 time=0.019 msec
154DEBUG: checkpoint sync: number=131 file=base/16385/68657_vm time=0.019 msec
155DEBUG: checkpoint sync: number=132 file=base/16762/77442 time=0.018 msec
156DEBUG: checkpoint sync: number=133 file=base/16762/69413 time=0.018 msec
157DEBUG: checkpoint sync: number=134 file=base/27036/68511 time=0.025 msec
158DEBUG: checkpoint sync: number=135 file=base/27036/68511_vm time=0.019 msec
159DEBUG: checkpoint sync: number=136 file=base/82382/83607 time=0.028 msec
160DEBUG: checkpoint sync: number=137 file=base/82382/83607_vm time=0.018 msec
161DEBUG: checkpoint sync: number=138 file=base/82382/83451 time=0.019 msec
162DEBUG: checkpoint sync: number=139 file=base/82382/83451.1 time=0.018 msec
163DEBUG: checkpoint sync: number=140 file=base/82382/83451.2 time=0.018 msec
164DEBUG: checkpoint sync: number=141 file=base/82382/83451.3 time=0.019 msec
165DEBUG: checkpoint sync: number=142 file=base/16762/77643 time=0.019 msec
166DEBUG: checkpoint sync: number=143 file=base/82382/83539 time=0.017 msec
167DEBUG: checkpoint sync: number=144 file=base/16762/77448 time=0.019 msec
168DEBUG: checkpoint sync: number=145 file=base/16762/77644 time=0.018 msec
169DEBUG: checkpoint sync: number=146 file=base/82382/82383 time=0.019 msec
170DEBUG: checkpoint sync: number=147 file=base/82382/83550 time=0.018 msec
171DEBUG: checkpoint sync: number=148 file=base/82382/83550_fsm time=0.018 msec
172DEBUG: checkpoint sync: number=149 file=base/27036/186067 time=1.065 msec
173DEBUG: checkpoint sync: number=150 file=base/82382/83444 time=0.021 msec
174DEBUG: checkpoint sync: number=151 file=base/82382/83540 time=0.019 msec
175DEBUG: checkpoint sync: number=152 file=base/82382/83540_fsm time=0.019 msec
176DEBUG: checkpoint sync: number=153 file=base/27036/68260 time=0.022 msec
177DEBUG: checkpoint sync: number=154 file=base/27036/68260_fsm time=0.019 msec
178DEBUG: checkpoint sync: number=155 file=base/27036/68260_vm time=0.019 msec
179DEBUG: checkpoint sync: number=156 file=base/16762/77900 time=0.019 msec
180DEBUG: checkpoint sync: number=157 file=base/16762/77897 time=0.019 msec
181DEBUG: checkpoint sync: number=158 file=base/16762/69414 time=0.018 msec
182DEBUG: checkpoint sync: number=159 file=base/82382/83661 time=0.018 msec
183DEBUG: checkpoint sync: number=160 file=base/82382/83661_fsm time=0.019 msec
184DEBUG: checkpoint sync: number=161 file=base/82382/83661_vm time=0.018 msec
185DEBUG: checkpoint sync: number=162 file=base/82382/82469 time=0.019 msec
186DEBUG: checkpoint sync: number=163 file=base/16762/77445 time=0.019 msec
187DEBUG: checkpoint sync: number=164 file=base/82382/83780 time=0.018 msec
188DEBUG: checkpoint sync: number=165 file=base/82382/83780_fsm time=0.019 msec
189DEBUG: checkpoint sync: number=166 file=base/82382/83780_vm time=0.018 msec
190DEBUG: checkpoint sync: number=167 file=base/27036/68514 time=0.019 msec
191DEBUG: checkpoint sync: number=168 file=base/27036/68514_fsm time=0.018 msec
192DEBUG: checkpoint sync: number=169 file=base/27036/68514_vm time=0.019 msec
193DEBUG: checkpoint sync: number=170 file=base/82382/83817 time=0.018 msec
194DEBUG: checkpoint sync: number=171 file=base/82382/83817_fsm time=0.028 msec
195DEBUG: checkpoint sync: number=172 file=base/82382/82422 time=0.018 msec
196DEBUG: checkpoint sync: number=173 file=base/82382/82422_fsm time=0.018 msec
197LOG: checkpoint complete: wrote 95206 buffers (18.2%); 0 transaction log file(s) added, 0 removed, 40 recycled; write=523.261 s, sync=0.314 s, total=524.054 s; sync files=173, lo
198ngest=0.289 s, average=0.001 s; distance=668789 kB, estimate=668789 kB
199DEBUG: MultiXactId wrap limit is 2147483648, limited by database with OID 27036
200DEBUG: oldest MultiXactId member is at offset 1
201DEBUG: MultiXact member stop limit is now 4294914944 based on MultiXact 1
202FATAL: database "pgsql" does not exist
203NOTICE: shutting down
204LOG: checkpoint starting: shutdown immediate
205DEBUG: performing replication slot checkpoint
206LOG: checkpoint complete: wrote 0 buffers (0.0%); 0 transaction log file(s) added, 0 removed, 40 recycled; write=0.015 s, sync=0.000 s, total=0.068 s; sync files=0, longest=0.000
207 s, average=0.000 s; distance=0 kB, estimate=601910 kB
208NOTICE: database system is shut down
209$
After this completed gracefully, starting the database with the normal process worked again.
For some reason, even though I used the intended process to shut things down, something got stuck/fell over w/o logging something, and then decided not to recover on startup.
Comments