[bug-mailutils] movemail 3.1, hanging again

[Jean Louis]

Hello Sergey,

I have immediately unpacked and tried the mailtuils 3.1, by thinking
movemail would get all the improvements from last days, but it is
hanging again.

I have tried using the patch, but it has got rejected, now I have
again non-functional version of movemail.

rejected:

--- libmailutils/stream/stream.c
+++ libmailutils/stream/stream.c
@@ -1160,7 +1160,7 @@ mu_stream_ioctl (mu_stream_t stream, int family, int opcode, void *ptr)
{
int rc;
_bootstrap_event (stream);
- if ((rc = _stream_flush_buffer (stream, _MU_STR_FLUSH_ALL)))
+ if ((rc = _stream_flush_buffer (stream, _MU_STR_FLUSH_ALL|_MU_STR_FLUSH_KEEP)))
return rc;
if (stream->ctl == NULL)
return ENOSYS;

And -u is anyway not working, I guess it searched for UIDL in Maildir,
and if it does not find, it stops operation. It shall be comparing
UIDLS of messages I assume.

Jean

[Sergey Poznyakoff]

Hi Jean,

Post by Jean Louis
I have immediately unpacked and tried the mailtuils 3.1, by thinking
movemail would get all the improvements from last days

As a matter of fact, there were no improvements to movemail since
November 4 (these are, of course incorporated in the release, albeit
entirely irrelevant in this case).

Post by Jean Louis
I have tried using the patch, but it has got rejected

That patch is no longer needed. The affected fragment of code had been
rewritten from scratch (3733cba7) with the proposed functionality included.

Post by Jean Louis
but it is hanging again.

As I already stated, to investigate this I need a complete session
transcript. Unfortunately you did not provide it and I cannot reproduce
this behavior either. Can you send me off-list the unabridged
transcript exactly as obtained using the 'mailbox.=prot' debug level?

Post by Jean Louis
And -u is anyway not working

I know the reason and have already got a plan of fixing it, although I
cannot, unfortunately, give any exact date when the fix will appear. I
estimate it to take a couple of weeks to finish.

Regards,
Sergey

[Jean Louis]

Hello,

I have tried it again with debugging now, and during debugging it did
not hang.

Once I encounter again that it hangs, I will send you debugging
output.

Jean

Post by Sergey Poznyakoff
As I already stated, to investigate this I need a complete session
transcript. Unfortunately you did not provide it and I cannot reproduce
this behavior either. Can you send me off-list the unabridged
transcript exactly as obtained using the 'mailbox.=prot' debug level?

Post by Jean Louis
And -u is anyway not working

I know the reason and have already got a plan of fixing it, although I
cannot, unfortunately, give any exact date when the fix will appear. I
estimate it to take a couple of weeks to finish.

[Jean Louis]

Post by Sergey Poznyakoff
Hi Jean,

movemail: S: * 40 FETCH (FLAGS (\Seen \Recent))
movemail: S: 84 OK FETCH completed.
movemail: number of processed messages: 40
movemail: number of errors: 0 / 0
########### HERE IS WHERE IT HANGS FOR MINUTES ################

That solves it. It did not hang at all. It busy incorporating changes
to your destination mailbox. There can be many reasons why it took so
long: slow disk access, huge mbox maildrop, disk problems, etc, etc.
Next time it occurs, flip to another console and check the status of the
running movemail with ps(1) or top(1). That will give more insight.

Here is again the new report, and maybe you should still see why it is
taking long. The 16 emails to be deleted, in this case of today,
required maybe 5 minutes, I mean I could prepare the tea, crush the
ginger, mix with sugar, put some mustard on the bread and hot dog, and
it still was not finished. Then I started eating breakfast, I was
watching with the ps the movemail, and I could not see any changes in
the command line, to get more insight.

Now, I am using other software to access my IMAP server, and no
software required more than 5 minutes to delete messages. So it is not
the IMAP server for sure that could be slow.

Also I am daily using my maildirs with other software, and my computer
is pretty fast, so I don't believe it is my hard disk or CPU, delaying
it.

And if delaying, like that long, than the debug output should say WHY
is it delaying, otherwise it is impossible to debug it, right?

It is really not usable if it delays for 5 minutes.

movemail: S: )
movemail: S: 36 OK FETCH completed.
movemail: number of processed messages: 16
movemail: number of errors: 0 / 0

########### HERE IS WHERE IT HANGS FOR MINUTES ################

movemail: C: 37 STORE 15 FLAGS.SILENT (\Deleted)
movemail: S: 37 OK STORE completed.
movemail: C: 38 STORE 5,11,16 FLAGS.SILENT (\Deleted \Seen)
movemail: S: 38 OK STORE completed.
movemail: C: 39 EXPUNGE
movemail: S: * 1 EXPUNGE
movemail: ignoring unrecognized response
movemail: S: * 1 EXPUNGE
movemail: ignoring unrecognized response
movemail: S: * 1 EXPUNGE
movemail: ignoring unrecognized response
movemail: S: * 1 EXPUNGE
movemail: ignoring unrecognized response

[Sergey Poznyakoff]

Post by Jean Louis
Here is again the new report, and maybe you should still see why it is
taking long.

The information you provide is unfortunately insufficient to make any
conclusions.

Post by Jean Louis
The 16 emails to be deleted, in this case of today, required maybe 5
minutes,

[...]

Post by Jean Louis
Now, I am using other software to access my IMAP server, and no
software required more than 5 minutes to delete messages. So it is not
the IMAP server for sure that could be slow.

Can you *please* make some more attention to what I am saying to you?

Post by Jean Louis

[Movemail is] busy incorporating changes to your destination mailbox.

In other words, there's no question about IMAP server or whatever. The
delay occurs while delivering data to your hard disk files. Is that
clear?

Post by Jean Louis
Then I started eating breakfast, I was
watching with the ps the movemail, and I could not see any changes in
the command line, to get more insight.

Jean, that's all very nice, but that's emotional. And the truth is:
emotions have very little value when it comes to technical decisions.
The problem we are facing is technical. So please let's put emotions
aside and try to get to the facts. It is quite obvious you didn't see
any changes in ps. The question is: what *was* the output of ps?

Regards,
Sergey

[Jean Louis]

Post by Sergey Poznyakoff
The information you provide is unfortunately insufficient to make any
conclusions.

If I could provide you, I would. I do not have in ~/Maildir any emails
when movemail is moving it. So it is empty all in cur/new/tmp.

My email reading method is to finish all emails, before moving new
mails, so each time I am moving emails, my inbox is completely
empty. No other application has any problems writing into that
Maildir, so it is not logical that movemail is delaying 5 minutes
incorporating changes to destination mailbox, and that it depends of
my system.

Maybe movemail is slow? But I cannot debug it, as I don't have that
possibility.

Post by Sergey Poznyakoff
Can you *please* make some more attention to what I am saying to you?

[Movemail is] busy incorporating changes to your destination mailbox.

In other words, there's no question about IMAP server or whatever. The
delay occurs while delivering data to your hard disk files. Is that
clear?

Sorry, not quite clear. When application requires 5 minutes to deliver
5-10 emails into my Maildir, which is empty, then whatever it is
underlying, like incorporating changes, it is a bug, as no changes
require 5 minutes.

Post by Sergey Poznyakoff
aside and try to get to the facts. It is quite obvious you didn't see
any changes in ps. The question is: what *was* the output of ps?

just the command line of the movemail, something like below.

The debug is always turned on.

If movemail requires 5 minutes to incorporate changes like you state,
it shall say so in the debug output exactly what it is doing. As
otherwise I cannot debug it.

Jean


(defparameter movemail
(concatenate 'string
"movemail --debug-level='mailbox.prot,!trace6' -v 'imaps://" username ":" password "@" server ":" port "/"
;; "movemail -v 'imaps://" username ":" password "@" server ":" port "/"
(car (gethash (intern mailbox) mailboxes)) "' \""
(cadr (gethash (intern mailbox) mailboxes))
"\""))

[Sergey Poznyakoff]

Post by Jean Louis
My email reading method is to finish all emails, before moving new
mails, so each time I am moving emails, my inbox is completely
empty. No other application has any problems writing into that
Maildir, so it is not logical that movemail is delaying 5 minutes
incorporating changes to destination mailbox, and that it depends of
my system.

It's pretty strange indeed.

Post by Jean Louis
Sorry, not quite clear. When application requires 5 minutes to deliver
5-10 emails into my Maildir, which is empty, then whatever it is
underlying, like incorporating changes, it is a bug,

It can't be ruled out, of course.

Post by Jean Louis
just the command line of the movemail, something like below.

Jean, the purpose of the ps is not to show the command line, as it is
known to you anyways. It's here to deliver exact information about the
state of the process. Let's be more precise then. Next time you
experience the bug, please:

1. Flip to another console and run

ps axwl | grep movemail

(exactly like that, don't omit nor add anything) and send me the output
it produces.

2. In this output, mark the PID of the process (see ps(1) if you don't
know where to find it) and then run

strace -p $PID 2> log

where $PID is the PID numeric value you snatched from the ps output.
Send me the produced log.

Regards,
Sergey

[Jean Louis]

Hello,

Before I can do the strace, I see the flag "wait_o" with the below
command.

Post by Sergey Poznyakoff
1. Flip to another console and run
ps axwl | grep movemail
(exactly like that, don't omit nor add anything) and send me the output
it produces.

0 1001 814 813 20 0 170108 11760 wait_o D+ tty1 0:10 movemail --debug-level=mailbox.prot,!trace6 -v imaps://username:***@server:993/INBOX /home/data1/protected/Maildir
0 1001 941 678 20 0 117836 3304 wait Ss ? 0:00 /bin/bash -c ps axwl | grep movemail
0 1001 943 941 20 0 114684 2128 pipe_w S ? 0:00 grep movemail

Post by Sergey Poznyakoff
2. In this output, mark the PID of the process (see ps(1) if you don't
know where to find it) and then run
strace -p $PID 2> log

I will do in next run, when it happens. Right now it is waiting with 6
messages for longer time. I could finish writing this email...

Jean Louis

[Sergey Poznyakoff]

Post by Jean Louis

Post by Sergey Poznyakoff
1. Flip to another console and run
ps axwl | grep movemail
(exactly like that, don't omit nor add anything) and send me the output
it produces.

That's all I wanted to know. The movemail process is in ininterruptible
sleep state (as indicated by the D) flag. This means it is waiting for
the disk operation to finish. Practically this rules out a possibility
that it is a bug.

Post by Jean Louis

Post by Sergey Poznyakoff
strace -p $PID 2> log

I will do in next run, when it happens. Right now it is waiting with 6
messages for longer time.

You should have done it right now since you had plenty of time.

Regards,
Sergey

[Jean Louis]

Hello Sergey,

Post by Sergey Poznyakoff

Post by Jean Louis

Post by Sergey Poznyakoff
1. Flip to another console and run
ps axwl | grep movemail
(exactly like that, don't omit nor add anything) and send me the output
it produces.

That's all I wanted to know. The movemail process is in ininterruptible
sleep state (as indicated by the D) flag. This means it is waiting for
the disk operation to finish. Practically this rules out a possibility
that it is a bug.

Post by Jean Louis

Post by Sergey Poznyakoff
strace -p $PID 2> log

I will do in next run, when it happens. Right now it is waiting with 6
messages for longer time.

You should have done it right now since you had plenty of time.

My computer was working, or was up, all the time. And by doing so, I
figured out, that problems with movemail, appear after the boot, in
first one or two batches. During the computer work, like last 24
hours, I have not experienced problems.

I did not look into the movemail source code, and I don't know C.

I am next time, on boot, going to use other Maildir as destination to
test it.

And I have to find strace package to install it.

Jean Louis

[Jean Louis]

Post by Sergey Poznyakoff

Post by Jean Louis
Sorry, not quite clear. When application requires 5 minutes to deliver
5-10 emails into my Maildir, which is empty, then whatever it is
underlying, like incorporating changes, it is a bug,

It can't be ruled out, of course.

You may be sure, that it is not about incorporating changes, as I can
just normally enter mutt in other window and read the Maildir.

When it says "number of processed messages", messages are nicely in
the Maildir, can be read, or deleted.

Jean