I want to play movies without hangs

[Alex Schuster]

Hi there!

Strange things are going on here.

I've written here in the past about my performance problems. My dual-core
had trouble playing movies without stuttering when there was I/O. It was
mainly swapping that caused this, and 8 G were not enough for me running
KDE4.

Then my hardware broke, and I got new one, except for the system hard
drive and the PSU. It's an AMD FX-4100 quad-core with 3.6 GHz, 16 G of
RAM. Running gentoo-sources-3.2.1 as kernel. But it seems playing movies
got even worse!

The videos do not need to have high quality. When I do this, I get
interruptions, sometimes for more than a whole second:

# dd if=/dev/zero of=/tmp/argh bs=10M count=1000

My whole system is encrypted, but the same happens with unencrypted
partitions. All are on LVM. When I write to another drive, there is no
effect. Throughput is around 50-60 MB/s.

Any ideas where to look? I think I'll create a completely fresh
kernel .config with genkernel, maybe my own .config has some weird
problem. But I tried similar things in the past already, getting a kernel
from a live cd, to no effect.

I put cache = 10240 into .mplayer/config to get 10 MB of video cached,
but I see no effect.

Playing music with Amarok is no problem.

My SATA drives are in AHCI mode, here's some dmesg info about that:

ahci 0000:00:11.0: version 3.0
ahci 0000:00:11.0: PCI INT A -> GSI 22 (level, low) -> IRQ 22
ahci 0000:00:11.0: AHCI 0001.0100 32 slots 4 ports 3 Gbps 0xf impl SATA
mode
ahci 0000:00:11.0: flags: 64bit ncq sntf ilck led clo pmp pio slum part
ccc sxs
scsi0 : ahci
scsi1 : ahci
scsi2 : ahci
scsi3 : ahci
ata1: SATA max UDMA/133 abar ***@0xff70b000 port 0xff70b100 irq 22
ata2: SATA max UDMA/133 abar ***@0xff70b000 port 0xff70b180 irq 22
ata3: SATA max UDMA/133 abar ***@0xff70b000 port 0xff70b200 irq 22
ata4: SATA max UDMA/133 abar ***@0xff70b000 port 0xff70b280 irq 22
ahci 0000:02:00.0: PCI INT A -> GSI 19 (level, low) -> IRQ 19
ahci 0000:02:00.0: irq 43 for MSI/MSI-X
ahci: SSS flag set, parallel bus scan disabled
ahci 0000:02:00.0: AHCI 0001.0200 32 slots 2 ports 6 Gbps 0x3 impl SATA
mode
ahci 0000:02:00.0: flags: 64bit ncq sntf stag led clo pmp pio slum part
ccc sxs
ahci 0000:02:00.0: setting latency timer to 64
scsi4 : ahci
scsi5 : ahci
ata5: SATA max UDMA/133 abar ***@0xff600000 port 0xff600100 irq 43
ata6: SATA max UDMA/133 abar ***@0xff600000 port 0xff600180 irq 43
pata_atiixp 0000:00:14.1: PCI INT A -> GSI 16 (level, low) -> IRQ 16
scsi6 : pata_atiixp
scsi7 : pata_atiixp
ata7: PATA max UDMA/100 cmd 0x1f0 ctl 0x3f6 bmdma 0xf000 irq 14
ata8: PATA max UDMA/100 cmd 0x170 ctl 0x376 bmdma 0xf008 irq 15

(ata7/8 is the additional PATA controller, seen with the pata_atiixp
driver. I have one drive there, but it is not being used.)

When my new girl-friend comes over and we want to watch a movie, and it
stutters... she will ask why I don't simply use Windows to get
better performance, her five year old PC would do this just fine. Wat do
I tell her? WHAT DO I TELL HER??


And then there's what happened yesterday. A world update was going on,
with libreoffice, firefox, wine and thunderbird emerging in parallel, all
big packages. I have the PORTAGE_TMPDIR on a 5GB tmpfs, only libreoffice
is being compiled on disk. Suddenly, my system became very unresponsive,
the mouse had disappeared, the KDE widgets did not update, and xosview
showed a load of 23. All 4 cores were at 100%, the type of usage was
io-wait. How can I find out in such a case which processes are waiting
for I/O? top showed nothing. The Ctrl-Esc task viewer of KDE showed some
processes being 'inactive on hard drive', does this men those are the
waiting tasks? They varied, they were mostly Akonadi stuff. I stopped
akonadi, and after a while the load dropped. But this may be a
coincidence.

After all had calmed down, I had 2G of swap in use. 16G total RAM, all
being used of course, but only 8G being needed according to the -/+
buffers/cache line in free -m, the other 8G are cache. Does my Linux
somehow prefer to have this much cache, even if tmpfs stuff gets put into
swap? I have vm.swappiness = 0 in /etc/sysctl.conf.

Is there a command to show me what processes the memory in swap belongs
to?

Wonko

[Mark Knecht]

Post by Alex Schuster
Hi there!

Hi back at ya.

Post by Alex Schuster
How can I find out in such a case which processes are waiting
for I/O? top showed nothing.

iotop is your friend.

I'll write more when I get some time to think

HTH,
Mark

[Alex Schuster]

Post by Mark Knecht

Post by Alex Schuster
How can I find out in such a case which processes are waiting
for I/O? top showed nothing.

iotop is your friend.

I had called it, but didn't spot the problem there. I don't remember
exactly what the output was, I had expected to see some process show a
large value in the IO column, but that was not the case. I THINK! Various
processes appeared, mostly Akonadi stuff.

I should have logged this, I don't remember this s well. I have caught
some illness, and had fever, which did not help my memory. There were
some kworker processes listed on top, but I don't remember whether in
iotop or in top.

Hmm. Now I just started Akonadi again, and get a lot I/O in iotop, and
xosview again shows much iowait CPU activity for a while. But that calmed
down after a minute.

Post by Mark Knecht
I'll write more when I get some time to think

Thanks :)

Wonko

[Mark Knecht]

Post by Alex Schuster

Post by Mark Knecht

Post by Alex Schuster
How can I find out in such a case which processes are waiting
for I/O? top showed nothing.

iotop is your friend.

I had called it, but didn't spot the problem there. I don't remember
exactly what the output was, I had expected to see some process show a
large value in the IO column, but that was not the case. I THINK! Various
processes appeared, mostly Akonadi stuff.
I should have logged this, I don't remember this s well. I have caught
some illness, and had fever, which did not help my memory. There were
some kworker processes listed on top, but I don't remember whether in
iotop or in top.
Hmm. Now I just started Akonadi again, and get a lot I/O in iotop, and
xosview again shows much iowait CPU activity for a while. But that calmed
down after a minute.

Post by Mark Knecht
I'll write more when I get some time to think

Thanks :)
       Wonko

Sorry. I was rushing then as now. If you start iotop and then hit the
'o' key it will show only processes actually doing io. If you're hang
is truly an iowait then my experience is that it should at least
identify what process is having the problem.

HTH,
Mark

[Paul Hartman]

Post by Alex Schuster
I've written here in the past about my performance problems. My dual-core
had trouble playing movies without stuttering when there was I/O. It was
mainly swapping that caused this, and 8 G were not enough for me running
KDE4.
Then my hardware broke, and I got new one, except for the system hard
drive and the PSU. It's an AMD FX-4100 quad-core with 3.6 GHz, 16 G of
RAM. Running gentoo-sources-3.2.1 as kernel. But it seems playing movies
got even worse!

You don't mention anything about video card or video driver setup.
That's the first thing I would suspect.

What video card? What drivers? Are you using hardware accelerated
movie playback?

[Alex Schuster]

Post by Paul Hartman

Post by Alex Schuster
Then my hardware broke, and I got new one, except for the system hard
drive and the PSU. It's an AMD FX-4100 quad-core with 3.6 GHz, 16 G of
RAM. Running gentoo-sources-3.2.1 as kernel. But it seems playing
movies got even worse!

You don't mention anything about video card or video driver setup.
That's the first thing I would suspect.
What video card? What drivers? Are you using hardware accelerated
movie playback?

Sorry. Radeon HD 4250 onboard graphics, using the open source radeon
driver. Hardware acceleration is working fine. As I wrote, it doesn't
matter which quality the videos are. There is not much CPU being used at
all, around 5% to 20%, so this is not the bottleneck.

Wonko

[Paul Hartman]

Post by Alex Schuster

Post by Paul Hartman

Post by Alex Schuster
Then my hardware broke, and I got new one, except for the system hard
drive and the PSU. It's an AMD FX-4100 quad-core with 3.6 GHz, 16 G of
RAM. Running gentoo-sources-3.2.1 as kernel. But it seems playing
movies got even worse!

You don't mention anything about video card or video driver setup.
That's the first thing I would suspect.
What video card? What drivers? Are you using hardware accelerated
movie playback?

Sorry. Radeon HD 4250 onboard graphics, using the open source radeon
driver. Hardware acceleration is working fine. As I wrote, it doesn't
matter which quality the videos are. There is not much CPU being used at
all, around 5% to 20%, so this is not the bottleneck.
       Wonko

I wonder if you copy the movie to /dev/shm first (so disk I/O is not
an issue) does it still have problems? At least this can potentially
eliminate disk I/O as the cause if something else weird is going on.
:)

For the problem of massive amounts of RAM consumed, that's strange.
Are you compiling debug symbols? That can make the RAM usage (in
linking especially) explode...

[Alex Schuster]

Post by Paul Hartman
I wonder if you copy the movie to /dev/shm first (so disk I/O is not
an issue) does it still have problems? At least this can potentially
eliminate disk I/O as the cause if something else weird is going on.
:)

Yes, this helps. As does copying the movie to another partition than that
on which I to the I/O with my dd if=/dev/zero of= command. If I dd to
this partition, tough, it happens again.

Post by Paul Hartman
For the problem of massive amounts of RAM consumed, that's strange.

It's been so for years... but with 16 G of RAM it's no longer an issue.
Well, unless this weird problem happened wth parallel emerges on tmpfs. It
doesn't happen every time though.

Post by Paul Hartman
Are you compiling debug symbols? That can make the RAM usage (in
linking especially) explode...

No. I sometimes enable it, but only when I hunt a bug and want to produce
better bug reports.
And even if the emerge would need very much memory, shouldn't this be
taken from the 8 G of caches being used, instead of starting to swap?

Wonko

[Walter Dnes]

Then my hardware broke, and I got new one...

I had ***EXACTLY THE SAME PROBLEM ON A FRESH INSTALL***. In My case
it was a 4+ year old Dell with onboard Intel GPU that was having
problems playing NHL Gamecenter Live streams at the slowest speed. I
solved the problem and sped up everything by doing...
1) emerge system
2) emerge world
3) rebuild the kernel and reboot

A fresh install will have the stage 3 binaries built with
lowest-common-denominator x86 or amd64 code (depending if you chose 32
or 64 bit install). This is necessary in order to allow the install
code to run on all CPUs with the target platform. The downside is that
you lose all the optimisations that make Gentoo scream. Rebuilding the
install as described above builds optimized (i.e. faster) binaries. My
CFLAGS line in /etc/make.conf is...

CFLAGS="-O2 -march=native -mfpmath=sse -fomit-frame-pointer -pipe"
CXXFLAGS="${CFLAGS}"

Before rebuilding your system, go over your USE flags to make sure
you've got the maximum optimization. To find out what your CPU
supports, execute the command

grep flags /proc/cpuinfo | head -1

This will define the limits what your system can support. For
instance, mplayer can use the following flags...

***@d530 ~ $ emerge -pv mplayer

These are the packages that would be merged, in order:

Calculating dependencies... done!
[ebuild R ] media-video/mplayer-1.0_rc4_p20110322-r1 USE="X a52 alsa ass dga encode gif jpeg mmx mmxext mng mp3 opengl png quicktime real rtmp sse sse2 ssse3 theora truetype win32codecs x264 xv xvid xvmc -3dnow -3dnowext -aalib (-altivec) -amr (-aqua) -bidi -bindist -bl -bluray -bs2b -cddb -cdio -cdparanoia -cpudetection -custom-cpuopts -debug -dirac -directfb -doc -dts -dv -dvb -dvd -dvdnav (-dxr3) -enca (-esd) -faac -faad -fbcon -ftp -ggi -gsm -iconv -ipv6 -jack -joystick -jpeg2k -ladspa -libcaca -libmpeg2 -lirc -live -lzo -mad -md5sum -mpg123 -nas -network -nut -openal -osdmenu -oss -pnm -pulseaudio -pvr -radio -rar -rtc -samba -schroedinger -sdl -shm -speex -tga -toolame -tremor -twolame -unicode -v4l -vdpau -vidix -vorbis -vpx -xanim -xinerama -xscreensaver -zoran" VIDEO_CARDS="
-mga -s3virge -tdfx -vesa" 0 kB

Your CPU will obviously support a different set of USE flags than
mine. Check the files /usr/portage/profiles/use.desc for a list of
global flags and /usr/portage/profiles/use.local.desc for
package-specific flags.

[Alex Schuster]

Post by Walter Dnes

Then my hardware broke, and I got new one...

I had ***EXACTLY THE SAME PROBLEM ON A FRESH INSTALL***. In My case
it was a 4+ year old Dell with onboard Intel GPU that was having
problems playing NHL Gamecenter Live streams at the slowest speed. I
solved the problem and sped up everything by doing...
1) emerge system
2) emerge world
3) rebuild the kernel and reboot

Good idea, Walter! But not in my case. The system had been set up long

Post by Walter Dnes
A fresh install will have the stage 3 binaries built with
lowest-common-denominator x86 or amd64 code (depending if you chose 32
or 64 bit install). This is necessary in order to allow the install
code to run on all CPUs with the target platform. The downside is that
you lose all the optimisations that make Gentoo scream. Rebuilding the
install as described above builds optimized (i.e. faster) binaries. My
CFLAGS line in /etc/make.conf is...
CFLAGS="-O2 -march=native -mfpmath=sse -fomit-frame-pointer -pipe"
CXXFLAGS="${CFLAGS}"

I had those, when I upgraded the hardware:
CFLAGS="-march=k8-sse3 -mfpmath=sse -O2 -pipe"
Or something very silimar.

But I also just did an emerge -e @world on the new system, using more
sophisticated CFLAGS. I got them by doing like suggested on [*], using
what -march=natve would do. And adding support for this graphite
stuff. They are:
CFLAGS="-pipe -march=amdfam10 -O2 \
-floop-interchange -floop-strip-mine -floop-block \
-msse -msse2 -msse3 -msse4 -msse4.1 -msse4.2 -m3dnow \
-mcx16 -msahf -maes -mpclmul -mpopcnt -mabm -mlwp -mavx \
--param l1-cache-size=16 --param l1-cache-line-size=64 \
--param l2-cache-size=2048"

Post by Walter Dnes
Before rebuilding your system, go over your USE flags to make sure
you've got the maximum optimization. To find out what your CPU
supports, execute the command
grep flags /proc/cpuinfo | head -1
This will define the limits what your system can support. For
instance, mplayer can use the following flags...
Calculating dependencies... done!
[ebuild R ] media-video/mplayer-1.0_rc4_p20110322-r1 USE="X a52
alsa ass dga encode gif jpeg mmx mmxext mng mp3 opengl png quicktime
real rtmp sse sse2 ssse3 theora truetype win32codecs x264 xv xvid xvmc
-3dnow -3dnowext -aalib (-altivec) -amr (-aqua) -bidi -bindist -bl
-bluray -bs2b -cddb -cdio -cdparanoia -cpudetection -custom-cpuopts
-debug -dirac -directfb -doc -dts -dv -dvb -dvd -dvdnav (-dxr3) -enca
(-esd) -faac -faad -fbcon -ftp -ggi -gsm -iconv -ipv6 -jack -joystick
-jpeg2k -ladspa -libcaca -libmpeg2 -lirc -live -lzo -mad -md5sum
-mpg123 -nas -network -nut -openal -osdmenu -oss -pnm -pulseaudio -pvr
-radio -rar -rtc -samba -schroedinger -sdl -shm -speex -tga -toolame
-tremor -twolame -unicode -v4l -vdpau -vidix -vorbis -vpx -xanim
-xinerama -xscreensaver -zoran" VIDEO_CARDS="-mga -s3virge -tdfx -vesa"
0 kB
Your CPU will obviously support a different set of USE flags than
mine. Check the files /usr/portage/profiles/use.desc for a list of
global flags and /usr/portage/profiles/use.local.desc for
package-specific flags.

These are my USE flags for mplayer, they should be fine:
[ebuild R ] media-video/mplayer-1.0_rc4_p20120213 USE="3dnow
3dnowext X a52 aalib alsa ass cdio dga directfb dts dv dvb dvd dvdnav
enca encode faad fbcon ftp ggi gif iconv ipv6 jack jpeg jpeg2k live mad
mmx mmxext mng mp3 nas network openal opengl osdmenu oss png pnm
quicktime rar real rtc samba sdl shm speex sse sse2 ssse3 theora toolame
tremor truetype twolame unicode vorbis x264 xinerama xscreensaver xv xvid
(-altivec) (-aqua) -bidi -bindist -bl -bluray -bs2b -cddb -cdparanoia
-cpudetection -debug -doc (-dxr3) (-esd) -faac -gsm -joystick -ladspa
-libcaca -libmpeg2 -lirc -lzo -md5sum -nut -pulseaudio -pvr -radio -rtmp
-tga -v4l -vdpau (-vidix) (-win32codecs) -xanim -xvmc -zoran"
VIDEO_CARDS="-mga -s3virge -tdfx" 0 kB

Now I'm bulding a new kernel, using genkernel, and without providing a
custom made .config. Just in case I have some weird setting somewhere
(debug output for SCSI stuff or something like that).

[later...]

So I did. Argh. I thought genkernel was smart enough to generate a
working kernel from scratch, if no existing .config would be given. But
the initramfs could not open my encrypted root partition, until I compiled
XTS and AES directly into the kernel, not only as modules. Genkernel did
not include modules for my NIC, somewhat annoying because I had to wait
several minutes for mysql to start, until I could open a root shell. KDM
was already running at that time, but I only saw a blank screen, because
the radeon stuff was not compiled with KMS. There's also something
wrong with my hardware clock. And iotop does not work, the kernel is
missing CONFIG_TASKSTATS, CONFIG_TASK_DELAY_ACCT
and CONFIG_TASK_IO_ACCOUNTING.

But now I have KDE running again. As soon as I do my dd command, mplayer
stutters, sometimes hanging for seconds. So it's probably not a flaw in
my kernel .config. Too bad, I hoped that was the problem.

I'm out of ideas now. This is really annoying, my system is fast, mplayer
is using around 20% only, the system is mostly idle, but when there is
I/O, videos do not run smoothly. And I do not want to copy every movie I
want to watch to another drive first.

BTW, when I do the dd if=/dev/zero of=/home/argh bs=10M count=1000
command, and interrupt with Ctrl-C, it takes some seconds until it stops.
Shouldn't this happen immediately, unless I specify a very large block
size? Writing speed is okay, 100 MB/s.

Time to go to sleep now.

[*] http://en.gentoo-wiki.com/wiki/Safe_Cflags#-march.3Dnative

Wonko

[James Broadhead]

Post by Alex Schuster

Then my hardware broke, and I got new one...

  I had ***EXACTLY THE SAME PROBLEM ON A FRESH INSTALL***.  In My case
it was a 4+ year old Dell with onboard Intel GPU that was having
problems playing NHL Gamecenter Live streams at the slowest speed.  I
solved the problem and sped up everything by doing...
1) emerge system
2) emerge world
3) rebuild the kernel and reboot

Good idea, Walter! But not in my case. The system had been set up long

  A fresh install will have the stage 3 binaries built with
lowest-common-denominator x86 or amd64 code (depending if you chose 32
or 64 bit install).  This is necessary in order to allow the install
code to run on all CPUs with the target platform.  The downside is that
you lose all the optimisations that make Gentoo scream.  Rebuilding the
install as described above builds optimized (i.e. faster) binaries.  My
CFLAGS line in /etc/make.conf is...
CFLAGS="-O2 -march=native -mfpmath=sse -fomit-frame-pointer -pipe"
CXXFLAGS="${CFLAGS}"

CFLAGS="-march=k8-sse3 -mfpmath=sse -O2 -pipe"
Or something very silimar.
sophisticated CFLAGS. I got them by doing like suggested on [*], using
what -march=natve would do. And adding support for this graphite
CFLAGS="-pipe -march=amdfam10 -O2 \
       -floop-interchange -floop-strip-mine -floop-block \
       -msse -msse2 -msse3 -msse4 -msse4.1 -msse4.2 -m3dnow \
       -mcx16 -msahf -maes -mpclmul -mpopcnt -mabm -mlwp -mavx \
       --param l1-cache-size=16 --param l1-cache-line-size=64 \
       --param l2-cache-size=2048"

  Before rebuilding your system, go over your USE flags to make sure
you've got the maximum optimization.  To find out what your CPU
supports, execute the command
grep flags /proc/cpuinfo | head -1
  This will define the limits what your system can support.  For
instance, mplayer can use the following flags...
Calculating dependencies... done!
[ebuild   R    ] media-video/mplayer-1.0_rc4_p20110322-r1  USE="X a52
alsa ass dga encode gif jpeg mmx mmxext mng mp3 opengl png quicktime
real rtmp sse sse2 ssse3 theora truetype win32codecs x264 xv xvid xvmc
-3dnow -3dnowext -aalib (-altivec) -amr (-aqua) -bidi -bindist -bl
-bluray -bs2b -cddb -cdio -cdparanoia -cpudetection -custom-cpuopts
-debug -dirac -directfb -doc -dts -dv -dvb -dvd -dvdnav (-dxr3) -enca
(-esd) -faac -faad -fbcon -ftp -ggi -gsm -iconv -ipv6 -jack -joystick
-jpeg2k -ladspa -libcaca -libmpeg2 -lirc -live -lzo -mad -md5sum
-mpg123 -nas -network -nut -openal -osdmenu -oss -pnm -pulseaudio -pvr
-radio -rar -rtc -samba -schroedinger -sdl -shm -speex -tga -toolame
-tremor -twolame -unicode -v4l -vdpau -vidix -vorbis -vpx -xanim
-xinerama -xscreensaver -zoran" VIDEO_CARDS="-mga -s3virge -tdfx -vesa"
0 kB
  Your CPU will obviously support a different set of USE flags than
mine.  Check the files /usr/portage/profiles/use.desc for a list of
global flags and /usr/portage/profiles/use.local.desc for
package-specific flags.

[ebuild   R    ] media-video/mplayer-1.0_rc4_p20120213  USE="3dnow
3dnowext X a52 aalib alsa ass cdio dga directfb dts dv dvb dvd dvdnav
enca encode faad fbcon ftp ggi gif iconv ipv6 jack jpeg jpeg2k live mad
mmx mmxext mng mp3 nas network openal opengl osdmenu oss png pnm
quicktime rar real rtc samba sdl shm speex sse sse2 ssse3 theora toolame
tremor truetype twolame unicode vorbis x264 xinerama xscreensaver xv xvid
(-altivec) (-aqua) -bidi -bindist -bl -bluray -bs2b -cddb -cdparanoia
-cpudetection -debug -doc (-dxr3) (-esd) -faac -gsm -joystick -ladspa
-libcaca -libmpeg2 -lirc -lzo -md5sum -nut -pulseaudio -pvr -radio -rtmp
-tga -v4l -vdpau (-vidix) (-win32codecs) -xanim -xvmc -zoran"
VIDEO_CARDS="-mga -s3virge -tdfx" 0 kB
Now I'm bulding a new kernel, using genkernel, and without providing a
custom made .config. Just in case I have some weird setting somewhere
(debug output for SCSI stuff or something like that).
[later...]
So I did. Argh. I thought genkernel was smart enough to generate a
working kernel from scratch, if no existing .config would be given. But
the initramfs could not open my encrypted root partition, until I compiled
XTS and AES directly into the kernel, not only as modules. Genkernel did
not include modules for my NIC, somewhat annoying because I had to wait
several minutes for mysql to start, until I could open a root shell. KDM
was already running at that time, but I only saw a blank screen, because
the radeon stuff was not compiled with KMS. There's also something
wrong with my hardware clock. And iotop does not work, the kernel is
missing CONFIG_TASKSTATS, CONFIG_TASK_DELAY_ACCT
and CONFIG_TASK_IO_ACCOUNTING.
But now I have KDE running again. As soon as I do my dd command, mplayer
stutters, sometimes hanging for seconds. So it's probably not a flaw in
my kernel .config. Too bad, I hoped that was the problem.
I'm out of ideas now. This is really annoying, my system is fast, mplayer
is using around 20% only, the system is mostly idle, but when there is
I/O, videos do not run smoothly. And I do not want to copy every movie I
want to watch to another drive first.
BTW, when I do the dd if=/dev/zero of=/home/argh bs=10M count=1000
command, and interrupt with Ctrl-C, it takes some seconds until it stops.
Shouldn't this happen immediately, unless I specify a very large block
size? Writing speed is okay, 100 MB/s.
Time to go to sleep now.
[*] http://en.gentoo-wiki.com/wiki/Safe_Cflags#-march.3Dnative
       Wonko

Please try:
~/.mplayer/config
lavdopts=threads=2
# Use 128MiB input cache by default.
cache = 131072
# Prefill 20% of the cache before starting playback.
cache-min = 20.0

Which should eliminate disk IO somewhat

[Alex Schuster]

Post by James Broadhead
~/.mplayer/config
lavdopts=threads=2
# Use 128MiB input cache by default.
cache = 131072
# Prefill 20% of the cache before starting playback.
cache-min = 20.0
Which should eliminate disk IO somewhat

James, thanks for your input. I already had threads = 2 in my config, now
I notice that this syntax is not valid. My cache setting was only 10M, and
I did not know about cache-min.

First, I thought these settings would help, the video I tested this with
this night no longer had problems. I commented the settings one after
another in order to find out which settings exactly helped most, and even
without them, it played fine. Maybe the whole video was in the cache at
that time already.

Then I tried another video, again with the settings you suggested, and it
stuttered. There were small pauses when the system did some stuff, and
when I did my dd test, the pauses were as long as five seconds. FIVE
SECONDS!

Wonko

[Urs Schutz]

On Sat, 18 Feb 2012 18:04:07 +0100
Alex Schuster <***@wonkology.org> wrote:

...

Post by Alex Schuster
Then I tried another video, again with the settings you
suggested, and it stuttered. There were small pauses when
the system did some stuff, and when I did my dd test, the
pauses were as long as five seconds. FIVE SECONDS!
Wonko

Just an idea: Is the disk OK? Replace /dev/sda with your
disk...

smartctl -t short /dev/sda

and after some minutes

smartctl --all /dev/sda

If all went OK then the status is «PASSED», and you could
try the extended or long tests with smartctl.
I had a bad disk here, which resulted in slow IO, but not
complete failure. Smart detected this immediately. Sorry,
I do not know how to check disks with LVM.

Urs

[Alex Schuster]

Post by Urs Schutz
Just an idea: Is the disk OK? Replace /dev/sda with your
disk...
smartctl -t short /dev/sda
and after some minutes
smartctl --all /dev/sda
If all went OK then the status is «PASSED», and you could
try the extended or long tests with smartctl.

I have smartd running. A short self test is done every day, and a long test
once per week.

Post by Urs Schutz
I had a bad disk here, which resulted in slow IO, but not
complete failure. Smart detected this immediately. Sorry,
I do not know how to check disks with LVM.

Didn't you get errors in yslog then?

I also thought about swapping the system drive - I have a larger backup
drive, with nearly identical logical volumes on it, where I make backups
with rdiffbackup. So even the content is identical, except for an additional
rdiff-backup directory containing the increments. So all I have to do is to
echange the two volume group names, reboot, and the system will run from the
other drive. But I very much doubt this will help, transfer speed looks okay
to me, around 100 MB/s with dd.

Wonko

[Alex Schuster]

Some while ago, I wrote:

[
mplayer stutters when I/O is going on, even hangs for seconds when I do a
dd if=/dev/zero of=somefile bs=1M
]

Post by Alex Schuster

Post by Urs Schutz
Just an idea: Is the disk OK? Replace /dev/sda with your
disk...

[...]

Post by Alex Schuster

Post by Urs Schutz
I had a bad disk here, which resulted in slow IO, but not
complete failure. Smart detected this immediately. Sorry,
I do not know how to check disks with LVM.

Didn't you get errors in yslog then?
I also thought about swapping the system drive - I have a larger backup
drive, with nearly identical logical volumes on it, where I make
backups with rdiffbackup. So even the content is identical, except for
an additional rdiff-backup directory containing the increments. So all
I have to do is to echange the two volume group names, reboot, and the
system will run from the other drive. But I very much doubt this will
help, transfer speed looks okay to me, around 100 MB/s with dd.

I did it in another way. I created a large file system (LVM) on my 2nd
drive, copied /, /usr, /var, /opt and /home over. My whole system is
encrypted, but I omitted this, just to make sure this is not the
bottleneck.

Alas, no change. Another thing I tried was to change the SATA mode in my
BIOS from AHCI to whatever the other option is. This did not help either.

Now this is really annoying. I watch small clips mostly, and can live
with that, and when I want to watch stuff with others, I copy the file to
tmpfs, which seems to help a lot.

But now I found another solution: NOT USING KDE.

When X crashed (trying to make the old Unreal game play), I fired up
another window manager, and when I played a video in there, there was no
problem at all. So, I have another workaround.

But does anyone have an idea, why running KDE is the problem? Disabling
desktop effects does not help.

I must be totally crazy because I still want to use KDE, despite the big
trouble it gives me nearly every day. Yes, most things work fine now, but
there are many many little problems, daily application crashes, and every
time I log in I fear that the desktop won't come up. 8G of RAM was not
enough to avoid swapping, so now I have 16G, that's fine, I no longer
care about kwin using 1G of my RAM. Oh, and I no longer use KMail, after
it ate thousands of mails I just wanted to move. No problem, they were
not important, but I no longer trust the KDEPIM suite. And it seems the
developers do not care about this, the bug report got no replies.

But anyway. Any idea why it only happens with KDE? I will ask on the KDE
mailing list, but I thought I post here first, maybe there's something
Gentoo-specific going on here.

Wonko

[Volker Armin Hemmann]

Post by Alex Schuster
[
mplayer stutters when I/O is going on, even hangs for seconds when I do a
dd if=/dev/zero of=somefile bs=1M
]

Post by Alex Schuster

Post by Urs Schutz
Just an idea: Is the disk OK? Replace /dev/sda with your
disk...

[...]

Post by Alex Schuster

Post by Urs Schutz
I had a bad disk here, which resulted in slow IO, but not
complete failure. Smart detected this immediately. Sorry,
I do not know how to check disks with LVM.

Didn't you get errors in yslog then?
I also thought about swapping the system drive - I have a larger backup
drive, with nearly identical logical volumes on it, where I make
backups with rdiffbackup. So even the content is identical, except for
an additional rdiff-backup directory containing the increments. So all
I have to do is to echange the two volume group names, reboot, and the
system will run from the other drive. But I very much doubt this will
help, transfer speed looks okay to me, around 100 MB/s with dd.

I did it in another way. I created a large file system (LVM) on my 2nd
drive, copied /, /usr, /var, /opt and /home over. My whole system is
encrypted, but I omitted this, just to make sure this is not the
bottleneck.
Alas, no change. Another thing I tried was to change the SATA mode in my
BIOS from AHCI to whatever the other option is. This did not help either.
Now this is really annoying. I watch small clips mostly, and can live
with that, and when I want to watch stuff with others, I copy the file to
tmpfs, which seems to help a lot.
But now I found another solution: NOT USING KDE.
When X crashed (trying to make the old Unreal game play), I fired up
another window manager, and when I played a video in there, there was no
problem at all. So, I have another workaround.
But does anyone have an idea, why running KDE is the problem? Disabling
desktop effects does not help.

nepomuk/virtuoso running in the background

whenever you have more than 1 process doing IO linux sucks ass.

Now, you write to a partition, nepomuk tries to index it (1 read) and your
player reading the video file (2nd read). Interactivity is shot.

Pause/kill nepomuk and look if it helps.

For me the worst case is: writing lots of files on a usb device - sucks
everywhere, even on a vt.

[Michael Hampicke]

Post by Volker Armin Hemmann

Post by Alex Schuster
[
mplayer stutters when I/O is going on, even hangs for seconds when I do a
dd if=/dev/zero of=somefile bs=1M
]

Post by Alex Schuster

Post by Urs Schutz
Just an idea: Is the disk OK? Replace /dev/sda with your
disk...

[...]

Post by Alex Schuster

Post by Urs Schutz
I had a bad disk here, which resulted in slow IO, but not
complete failure. Smart detected this immediately. Sorry,
I do not know how to check disks with LVM.

Didn't you get errors in yslog then?
I also thought about swapping the system drive - I have a larger backup
drive, with nearly identical logical volumes on it, where I make
backups with rdiffbackup. So even the content is identical, except for
an additional rdiff-backup directory containing the increments. So all
I have to do is to echange the two volume group names, reboot, and the
system will run from the other drive. But I very much doubt this will
help, transfer speed looks okay to me, around 100 MB/s with dd.

I did it in another way. I created a large file system (LVM) on my 2nd
drive, copied /, /usr, /var, /opt and /home over. My whole system is
encrypted, but I omitted this, just to make sure this is not the
bottleneck.
Alas, no change. Another thing I tried was to change the SATA mode in my
BIOS from AHCI to whatever the other option is. This did not help either.
Now this is really annoying. I watch small clips mostly, and can live
with that, and when I want to watch stuff with others, I copy the file to
tmpfs, which seems to help a lot.
But now I found another solution: NOT USING KDE.
When X crashed (trying to make the old Unreal game play), I fired up
another window manager, and when I played a video in there, there was no
problem at all. So, I have another workaround.
But does anyone have an idea, why running KDE is the problem? Disabling
desktop effects does not help.

nepomuk/virtuoso running in the background
whenever you have more than 1 process doing IO linux sucks ass.
Now, you write to a partition, nepomuk tries to index it (1 read) and your
player reading the video file (2nd read). Interactivity is shot.
Pause/kill nepomuk and look if it helps.
For me the worst case is: writing lots of files on a usb device - sucks
everywhere, even on a vt.

Maybe changing the kernel io scheduler will help?

# /usr/src/linux/Documentation/block/switching-sched.txt
# /usr/src/linux/Documentation/block/deadline-iosched.txt
# /usr/src/linux/Documentation/block/cfq-iosched.txt

[Volker Armin Hemmann]

Post by Michael Hampicke
Maybe changing the kernel io scheduler will help?
# /usr/src/linux/Documentation/block/switching-sched.txt
# /usr/src/linux/Documentation/block/deadline-iosched.txt
# /usr/src/linux/Documentation/block/cfq-iosched.txt

nice idea - but that didn't help in the past. Why should it help now?

The question is - why does disk IO make the mouse jerky and delays keyboard
input? That is just idiotic.

[Michael Hampicke]

Post by Volker Armin Hemmann
nice idea - but that didn't help in the past. Why should it help now?
The question is - why does disk IO make the mouse jerky and delays keyboard
input? That is just idiotic.

It was just an idea, but maybe there's something wrong on the hardware
side? Broken cable, hard drive about to die?

Have to checked the SMART data of your hard drive lately (error log,
relocate sector count). You can also run some tests with smartctl.

Or monitor your IO with dstat or iotop. iotop tells you which app causes
high io load.

Or maybe you kernel is using some generic and slow driver for your ata
controller instead of an optimized one?

Personally I never had IO problems on linux unless theres was something
wrong with either the hardware or the kernel driver.

Hope that helps.

[Volker Armin Hemmann]

Post by Michael Hampicke

Post by Volker Armin Hemmann
nice idea - but that didn't help in the past. Why should it help now?
The question is - why does disk IO make the mouse jerky and delays keyboard
input? That is just idiotic.

It was just an idea, but maybe there's something wrong on the hardware
side? Broken cable, hard drive about to die?

no, and this problem has been there since Suse 6.2 and kernel 2.2

Post by Michael Hampicke
Have to checked the SMART data of your hard drive lately (error log,
relocate sector count). You can also run some tests with smartctl.

of course

Post by Michael Hampicke
Or monitor your IO with dstat or iotop. iotop tells you which app causes
high io load.

cp of course

Post by Michael Hampicke
Or maybe you kernel is using some generic and slow driver for your ata
controller instead of an optimized one?

seriously...

Post by Michael Hampicke
Personally I never had IO problems on linux unless theres was something
wrong with either the hardware or the kernel driver.

good for you

Post by Michael Hampicke
Hope that helps.

no

[walt]

Post by Volker Armin Hemmann

Post by Michael Hampicke
Maybe changing the kernel io scheduler will help?
# /usr/src/linux/Documentation/block/switching-sched.txt
# /usr/src/linux/Documentation/block/deadline-iosched.txt
# /usr/src/linux/Documentation/block/cfq-iosched.txt

nice idea - but that didn't help in the past. Why should it help now?
The question is - why does disk IO make the mouse jerky and delays keyboard
input? That is just idiotic.

Have you tested ck-sources for comparison?

[Volker Armin Hemmann]

Post by walt

Post by Volker Armin Hemmann

Post by Michael Hampicke
Maybe changing the kernel io scheduler will help?
# /usr/src/linux/Documentation/block/switching-sched.txt
# /usr/src/linux/Documentation/block/deadline-iosched.txt
# /usr/src/linux/Documentation/block/cfq-iosched.txt

nice idea - but that didn't help in the past. Why should it help now?
The question is - why does disk IO make the mouse jerky and delays keyboard
input? That is just idiotic.

Have you tested ck-sources for comparison?

in the past, yes. Complete disappointment.

But it is ok, if the io happens between two very fast drives (ssd - ssd or ssd
- raid5). So the problem is:
io with one slow drive being written to
swapping

those cases are just horrible.

And vanilla-sources just work. Stable. Even with suspend to ram and ati
drivers.

[walt]

Post by Volker Armin Hemmann
And vanilla-sources just work. Stable. Even with suspend to ram and ati
drivers.

That confuses me. Are you saying the gentoo-sources don't 'just work'?

[Volker Armin Hemmann]

Post by walt

Post by Volker Armin Hemmann
And vanilla-sources just work. Stable. Even with suspend to ram and ati
drivers.

That confuses me. Are you saying the gentoo-sources don't 'just work'?

I don't know, haven't used gentoo sources in ages. I don't need fb
beautification, zcache, alps or livecd goodies. I like to be as close to
upstream as possible, so bug reports are taken seriously.

vanilla-sources work reliably with that annoying io-stuff. But no matter which
kernel I tried, ck, zen, gentoo, love etc pp that annoying io-stuff was always
there.. so no reason at all to use a patched up kernel.

[Alex Schuster]

Post by Michael Hampicke

Post by Volker Armin Hemmann

Post by Alex Schuster
[
mplayer stutters when I/O is going on, even hangs for seconds when I
do a dd if=/dev/zero of=somefile bs=1M
]

[...]

Post by Michael Hampicke

Post by Volker Armin Hemmann

Post by Alex Schuster
But now I found another solution: NOT USING KDE.
When X crashed (trying to make the old Unreal game play), I fired up
another window manager, and when I played a video in there, there
was no problem at all. So, I have another workaround.
But does anyone have an idea, why running KDE is the problem?
Disabling desktop effects does not help.

nepomuk/virtuoso running in the background
whenever you have more than 1 process doing IO linux sucks ass.
Now, you write to a partition, nepomuk tries to index it (1 read) and
your player reading the video file (2nd read). Interactivity is shot.
Pause/kill nepomuk and look if it helps.

Nice shot, but this cannot be the problem. Sorry, now that I read my post
again I see I did not mention that I _still_ have KDE running all the
time, doing whatever it does. I only need to play the videos outside in a
window manager I have running in parallel.
And the output of my dd action goes to another partition anyway, which is
not being indexed. I have 280,000 files indexed, a plain locate gives
nearly ten times as much.
Virtuoso has been a big pain in the past indeed. So I had it disabled
until KDE 4.8 I think. It does some indexing now when I log in, and it is
still doing so 8 hours after I logged in the last time, but I do not
notice this much, since KDE 4.8. Whether it affects my video playback I'm
not so sure, but I have the playback problems even when it is not running
- there's too much stuff going on all the time.

Post by Michael Hampicke

Post by Volker Armin Hemmann
For me the worst case is: writing lots of files on a usb device -
sucks everywhere, even on a vt.

Indeed. But should this affect an mplayer using a huge cache?
Does it also suck when you are writing with cp, using ionice -c 3?

Post by Michael Hampicke
Maybe changing the kernel io scheduler will help?
# /usr/src/linux/Documentation/block/switching-sched.txt
# /usr/src/linux/Documentation/block/deadline-iosched.txt
# /usr/src/linux/Documentation/block/cfq-iosched.txt

Nope. I just tried noop and deadline (cfq is my default) and did not see
much of a difference.

Wonko

[Mark Knecht]

Post by Alex Schuster
[
mplayer stutters when I/O is going on, even hangs for seconds when I do a
dd if=/dev/zero of=somefile bs=1M
]

Post by Alex Schuster

Post by Urs Schutz
Just an idea: Is the disk OK? Replace /dev/sda with your
disk...

[...]

Post by Alex Schuster

Post by Urs Schutz
I had a bad disk here, which resulted in slow IO, but not
complete failure. Smart detected this immediately. Sorry,
I do not know how to check disks with LVM.

Didn't you get errors in yslog then?
I also thought about swapping the system drive - I have a larger backup
drive, with nearly identical logical volumes on it, where I make
backups with rdiffbackup. So even the content is identical, except for
an additional rdiff-backup directory containing the increments. So all
I have to do is to echange the two volume group names, reboot, and the
system will run from the other drive. But I very much doubt this will
help, transfer speed looks okay to me, around 100 MB/s with dd.

I did it in another way. I created a large file system (LVM) on my 2nd
drive, copied /, /usr, /var, /opt and /home over. My whole system is
encrypted, but I omitted this, just to make sure this is not the
bottleneck.
Alas, no change. Another thing I tried was to change the SATA mode in my
BIOS from AHCI to whatever the other option is. This did not help either.
Now this is really annoying. I watch small clips mostly, and can live
with that, and when I want to watch stuff with others, I copy the file to
tmpfs, which seems to help a lot.
But now I found another solution: NOT USING KDE.
When X crashed (trying to make the old Unreal game play), I fired up
another window manager, and when I played a video in there, there was no
problem at all. So, I have another workaround.
But does anyone have an idea, why running KDE is the problem? Disabling
desktop effects does not help.
I must be totally crazy because I still want to use KDE, despite the big
trouble it gives me nearly every day. Yes, most things work fine now, but
there are many many little problems, daily application crashes, and every
time I log in I fear that the desktop won't come up. 8G of RAM was not
enough to avoid swapping, so now I have 16G, that's fine, I no longer
care about kwin using 1G of my RAM. Oh, and I no longer use KMail, after
it ate thousands of mails I just wanted to move. No problem, they were
not important, but I no longer trust the KDEPIM suite. And it seems the
developers do not care about this, the bug report got no replies.
But anyway. Any idea why it only happens with KDE? I will ask on the KDE
mailing list, but I thought I post here first, maybe there's something
Gentoo-specific going on here.
       Wonko

Hey Wonko,
OK, fire up two terminals. In one run top, hit 1 & z so you see all
your CPUs and then watch CPU usage. In the second terminal su to root
and run iotop -o. Now, watch for a few minutes and get a feel for
what's going on when video is not running. Then start your video and
watch IO usage and CPU usage. Where's the problem?

Once you get an idea where the bottleneck is we can address what a
solution might be. In general, if the CPUs aren't maxed out and it's
an I/O problem then usually a bit more buffering is a simple solution.
Other more draconian solution might be a real-time kernel with a
player (if there is one) that is set up for real-time playback.

Looking forward to hearing your test results.

Cheers,
Mark

[Alex Schuster]

Post by Mark Knecht

Post by Alex Schuster
[
mplayer stutters when I/O is going on, even hangs for seconds when I
do a dd if=/dev/zero of=somefile bs=1M
]

[...]

Post by Mark Knecht
OK, fire up two terminals. In one run top, hit 1 & z so you see all
your CPUs and then watch CPU usage. In the second terminal su to root
and run iotop -o. Now, watch for a few minutes and get a feel for
what's going on when video is not running. Then start your video and
watch IO usage and CPU usage. Where's the problem?
Once you get an idea where the bottleneck is we can address what a
solution might be. In general, if the CPUs aren't maxed out and it's
an I/O problem then usually a bit more buffering is a simple solution.
Other more draconian solution might be a real-time kernel with a
player (if there is one) that is set up for real-time playback.
Looking forward to hearing your test results.

Thanks for your support, Mark!

I did this already, but sometimes I do not notice anything. I guess it's
short I/O operations in that case. CPU load is not the problem, and it
happens for both high-quality videos and small ones.
Currently iotop shows stuff like kjournald, kworker, kdeinit4,
akonadiserver, firefox. And lots of virtuoso-t and nepomuk when I enable
indexing again, which I just suspended.
And mplayer of course, it shows up in about every 2nd redisplay, which
happens every second.

Well... but when I do the same in the other window manager, it seems I
see fewer processes then. Are they mostly suspended when I am on another
display?
And I should fire up the same stuff (Firefox, Chromium, maybe KDEPIM
stuff) in the other WM and see if this makes things worse. But I'll do
this tomorrow. Thanks for the inspiration, though, at least I have
something more to try now.

The interrupts are very small normally, but noticeable, annoying and
somewhat embarrassing. When they just happened I only noticed akonadi and
kjournald during that time. I can force larger interrupts by doing my dd
command.

But anyway - my intention is not so much to find out what all these
I/O processes are and how to make them calm down, renice them or whatever.
Four cores @ 3.6 GHz just should be able to play movies without any
interruption. And it _is_ possible, when I start the playback on another
window manager, while KDE is still running on the other display.

I could just switch to, um, ummmm.... Gnome maybe... or Xfce4... or
something else, but I would not like to do so. Despite by big KDE
problems. I hate KDE. But I still want it. I feel mad.

Wonko

[Alex Schuster]

Post by Alex Schuster

Post by Mark Knecht
OK, fire up two terminals. In one run top, hit 1 & z so you see all
your CPUs and then watch CPU usage. In the second terminal su to root
and run iotop -o. Now, watch for a few minutes and get a feel for
what's going on when video is not running. Then start your video and
watch IO usage and CPU usage. Where's the problem?
Once you get an idea where the bottleneck is we can address what a
solution might be. In general, if the CPUs aren't maxed out and it's
an I/O problem then usually a bit more buffering is a simple solution.
Other more draconian solution might be a real-time kernel with a
player (if there is one) that is set up for real-time playback.
Looking forward to hearing your test results.

Thanks for your support, Mark!
I did this already, but sometimes I do not notice anything. I guess it's
short I/O operations in that case. CPU load is not the problem, and it
happens for both high-quality videos and small ones.
Currently iotop shows stuff like kjournald, kworker, kdeinit4,
akonadiserver, firefox. And lots of virtuoso-t and nepomuk when I enable
indexing again, which I just suspended.
And mplayer of course, it shows up in about every 2nd redisplay, which
happens every second.
Well... but when I do the same in the other window manager, it seems I
see fewer processes then. Are they mostly suspended when I am on another
display?

I watched for longer now, and this does not seem to be true.

Post by Alex Schuster
And I should fire up the same stuff (Firefox, Chromium, maybe KDEPIM
stuff) in the other WM and see if this makes things worse. But I'll do
this tomorrow. Thanks for the inspiration, though, at least I have
something more to try now.

I am running Enlightenment 0.16 in parallel now, with Firefox, Chromium,
Kontact, Claws, Liferea, Amarok (which is doing a lot of I/OP stuff at
the moment according to iotop), and Dolphin showing a large directory of
multimedia files wit thumbnails. But I don't see akonadi related processes
in iotop, that is unusual.
I did the dd command to create more I/O. No gaps in video display at all.

When I play the video from within KDE (running Konsoles, Konqueror,
Dolphin and a lot of plasma stuff), I have gaps, and when I do the dd
command, there are in the range of seconds. Even for some seconds after
I canceled the dd.

I also tried a fresh, unconfigured KDE session by another user. I've
already done that, and there were also gaps in video playback, although
it seems they were fewer. But this time, I was not able to reproduce
them. Huh?

I guess I could remove anything running on my KDE desktop one by one,
including plasmoids, and see if playback gets better. But not now, I
finally have to actually do some work.

Wonko

[Neil Bothwick]

Post by Alex Schuster
I guess I could remove anything running on my KDE desktop one by one,
including plasmoids, and see if playback gets better. But not now, I
finally have to actually do some work.

I recently experienced slowdowns and delays with KDE. It turned out I had
inadvertently disabled swap (I'd rearranged my partitions and not updated
fstab). As soon as I gave it some swap space the delays disappeared.

[Alex Schuster]

Post by Neil Bothwick

Post by Alex Schuster
I guess I could remove anything running on my KDE desktop one by one,
including plasmoids, and see if playback gets better. But not now, I
finally have to actually do some work.

I recently experienced slowdowns and delays with KDE. It turned out I
had inadvertently disabled swap (I'd rearranged my partitions and not
updated fstab). As soon as I gave it some swap space the delays
disappeared.

There's plenty of swap space available. With 16 G of RAM it should not
be needed, but sometimes my load gets really really high, and when I can
use the system again, there is 2-3 G of swap usage. I haven't found out
yet what this is, it seems to happen when emerging things, maybe related
to having 5 G tmpfs for portage, but when it happened the last time only
100 M were being used.

Wonko

[Dale]

Post by Alex Schuster

Post by Neil Bothwick

Post by Alex Schuster
I guess I could remove anything running on my KDE desktop one by one,
including plasmoids, and see if playback gets better. But not now, I
finally have to actually do some work.

I recently experienced slowdowns and delays with KDE. It turned out I
had inadvertently disabled swap (I'd rearranged my partitions and not
updated fstab). As soon as I gave it some swap space the delays
disappeared.

There's plenty of swap space available. With 16 G of RAM it should not
be needed, but sometimes my load gets really really high, and when I can
use the system again, there is 2-3 G of swap usage. I haven't found out
yet what this is, it seems to happen when emerging things, maybe related
to having 5 G tmpfs for portage, but when it happened the last time only
100 M were being used.
Wonko

Is there a way to find out what is using swap? Maybe something related
to the video is on swap which at times can be slow, certainly slower
than ram.

I have always wondered how to find this out myself.

Dale

:-) :-)

[Adam Carter]

Is there a way to find out what is using swap?  Maybe something related
to the video is on swap which at times can be slow, certainly slower
than ram.
I have always wondered how to find this out myself.

Well the OS uses swap, i dont know if its possible to then tie that
directly to a process. You can find out if swap is being at all using
with vmstat; so= swap out, si=swap in.

For example, watch the following when you view the video

***@proxy ~ $ vmstat -S M 3
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
0 0 0 1290 379 6617 0 0 26 56 108 107 2 3 93 2
0 0 0 1290 379 6617 0 0 1 15 87 91 0 0 100 0
0 0 0 1290 379 6617 0 0 0 0 59 54 0 0 100 0
0 0 0 1290 379 6617 0 0 0 7 72 73 0 0 100 0

[Dale]

Post by Adam Carter

Post by Dale
Is there a way to find out what is using swap? Maybe something related
to the video is on swap which at times can be slow, certainly slower
than ram.
I have always wondered how to find this out myself.

Well the OS uses swap, i dont know if its possible to then tie that
directly to a process. You can find out if swap is being at all using
with vmstat; so= swap out, si=swap in.
For example, watch the following when you view the video
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
0 0 0 1290 379 6617 0 0 26 56 108 107 2 3 93 2
0 0 0 1290 379 6617 0 0 1 15 87 91 0 0 100 0
0 0 0 1290 379 6617 0 0 0 0 59 54 0 0 100 0
0 0 0 1290 379 6617 0 0 0 7 72 73 0 0 100 0

I'm not the OP but posted a two part post that didn't quite come out
like I expected. One for the OP to try to find out what was using swap,
just in case it mattered. Two to ask how that is done in case he didn't
know and for me since I don't know either. lol

I was hoping for a command that says program foo is using X amount of
swap but I guess not.

Interesting that there isn't a real good tool for this.

Dale

:-) :-)

[Alex Schuster]

Post by Dale
Is there a way to find out what is using swap? Maybe something related
to the video is on swap which at times can be slow, certainly slower
than ram.
I have always wondered how to find this out myself.

Me too, so when I had this sudden swap problem for the first time, I
searched for a method to do this and found a script here:
http://northernmost.org/blog/find-out-what-is-using-your-swap/

There's lots of information for all processes in /proc/<pid>/. Trying to
read /proc/<pid>/mem (I think it was this file) in mc was not such a good
idea, the system froze with lots of HD activity, and after half an hour I
rebooted with Alt-SysRq-{K,E,I,S,U,B}.

I improved the script a little, it allows sorting by PID, size and name,
and can restrict the output to specific processes or show only those
using more swap than specified. If interested you can download it here:
http://www.wonkology.org/utils/getswap
You need to be root to see processes you do not own.

But of course, I forgot to run it after the sudden swap problem happened
lately. So I still do not know what was going on there. I'll wait for the
next time it happens.

Wonko

[Michael Mol]

Post by Alex Schuster

Is there a way to find out what is using swap?  Maybe something related
to the video is on swap which at times can be slow, certainly slower
than ram.
I have always wondered how to find this out myself.

Me too, so when I had this sudden swap problem for the first time, I
http://northernmost.org/blog/find-out-what-is-using-your-swap/
There's lots of information for all processes in /proc/<pid>/. Trying to
read /proc/<pid>/mem (I think it was this file) in mc was not such a good
idea, the system froze with lots of HD activity, and after half an hour I
rebooted with Alt-SysRq-{K,E,I,S,U,B}.
I improved the script a little, it allows sorting by PID, size and name,
and can restrict the output to specific processes or show only those
http://www.wonkology.org/utils/getswap
You need to be root to see processes you do not own.
But of course, I forgot to run it after the sudden swap problem happened
lately. So I still do not know what was going on there. I'll wait for the
next time it happens.
       Wonko

sys-process/htop

[Alex Schuster]

Post by Michael Mol

Post by Alex Schuster

Is there a way to find out what is using swap?  Maybe something
related to the video is on swap which at times can be slow,
certainly slower than ram.
I have always wondered how to find this out myself.

Me too, so when I had this sudden swap problem for the first time, I
http://northernmost.org/blog/find-out-what-is-using-your-swap/
There's lots of information for all processes in /proc/<pid>/. Trying
to read /proc/<pid>/mem (I think it was this file) in mc was not such
a good idea, the system froze with lots of HD activity, and after
half an hour I rebooted with Alt-SysRq-{K,E,I,S,U,B}.
I improved the script a little, it allows sorting by PID, size and
name, and can restrict the output to specific processes or show only
those using more swap than specified. If interested you can download
it here: http://www.wonkology.org/utils/getswap
You need to be root to see processes you do not own.
But of course, I forgot to run it after the sudden swap problem
happened lately. So I still do not know what was going on there. I'll
wait for the next time it happens.
       Wonko

sys-process/htop

Huh? I only see the total amount of swap being used, but no entry per
process.

Wonko

[Michael Mol]

Post by Alex Schuster

Post by Michael Mol

Post by Alex Schuster

Is there a way to find out what is using swap?  Maybe something
related to the video is on swap which at times can be slow,
certainly slower than ram.
I have always wondered how to find this out myself.

Me too, so when I had this sudden swap problem for the first time, I
http://northernmost.org/blog/find-out-what-is-using-your-swap/
There's lots of information for all processes in /proc/<pid>/. Trying
to read /proc/<pid>/mem (I think it was this file) in mc was not such
a good idea, the system froze with lots of HD activity, and after
half an hour I rebooted with Alt-SysRq-{K,E,I,S,U,B}.
I improved the script a little, it allows sorting by PID, size and
name, and can restrict the output to specific processes or show only
those using more swap than specified. If interested you can download
it here: http://www.wonkology.org/utils/getswap
You need to be root to see processes you do not own.
But of course, I forgot to run it after the sudden swap problem
happened lately. So I still do not know what was going on there. I'll
wait for the next time it happens.
       Wonko

sys-process/htop

Huh? I only see the total amount of swap being used, but no entry per
process.

Hit F2, and go down to 'columns'. Anything per-process found under
/proc can be added as a column.

[Alex Schuster]

[about showing which processes use how much swap]
[...]

Post by Michael Mol

Post by Alex Schuster

Post by Michael Mol
sys-process/htop

Huh? I only see the total amount of swap being used, but no entry per
process.

Hit F2, and go down to 'columns'. Anything per-process found under
/proc can be added as a column.

Whoa! This is amazing, I did not know that htop can do all this. Thanks!

But I still cannot get it to display the swap used by processes. When I
add NSWAP and CNSWAP columns, they are not displayed. I found some
information on that, looks to me like this is not really supported:

http://stackoverflow.com/questions/479953/how-to-find-out-which-processes-are-swapping-in-linux
http://wiki.directi.com/display/tu/Understanding+Processes+in+Linux

Wonko

[Mark Knecht]

<SNIP>

Post by Michael Mol

Post by Alex Schuster

Post by Michael Mol
sys-process/htop

Huh? I only see the total amount of swap being used, but no entry per
process.

Hit F2, and go down to 'columns'. Anything per-process found under
/proc can be added as a column.
--
:wq

Anything in there show network through-put per process? I've been
looking for a way to monitor what's going to each of my VMs?

Cheers,
Mark

[Michael Mol]

Post by Mark Knecht
<SNIP>

Post by Michael Mol

Post by Alex Schuster

Post by Michael Mol
sys-process/htop

Huh? I only see the total amount of swap being used, but no entry per
process.

Hit F2, and go down to 'columns'. Anything per-process found under
/proc can be added as a column.

Anything in there show network through-put per process? I've been
looking for a way to monitor what's going to each of my VMs?

NAFAIK. Though if you get a kernel patch that gets per-process socket
auditing added, then it should show up. :)

I usually use iftop for watching flows. There's another tool I
installed which handles some things (such as IPv6) better, but inara
and kaylee are still down, so I can't peek at their world files to
find out what it was.

[Mark Knecht]

Post by Michael Mol

Post by Mark Knecht
<SNIP>

Post by Michael Mol

Post by Alex Schuster

Post by Michael Mol
sys-process/htop

Huh? I only see the total amount of swap being used, but no entry per
process.

Hit F2, and go down to 'columns'. Anything per-process found under
/proc can be added as a column.

Anything in there show network through-put per process? I've been
looking for a way to monitor what's going to each of my VMs?

NAFAIK. Though if you get a kernel patch that gets per-process socket
auditing added, then it should show up. :)
I usually use iftop for watching flows. There's another tool I
installed which handles some things (such as IPv6) better, but inara
and kaylee are still down, so I can't peek at their world files to
find out what it was.
--
:wq

Thanks. iftop is interesting but seems more focused on the provider of
the media source and less on the sink. I also use nettop to watch
overall bitrates but I suspect you have that one also.

Assume I have 3 VMs running and they are all streaming media.
VM1->Netflix, VM2->Hulu, VM3->Amazon, etc. What I'm really interested
in is something that would tell me how much bandwidth each VM is
getting. Per-process would almost certainly do that, and maybe that's
what I'll eventually have to do, but I'm hoping to find some little
app that maybe someone has put together.

Thanks,
Mark

[Alex Schuster]

Post by Mark Knecht
Anything in there show network through-put per process? I've been
looking for a way to monitor what's going to each of my VMs?

net-analyzer/nethogs does that.

Wonko

[Adam Carter]

Post by Alex Schuster
There's plenty of swap space available. With 16 G of RAM it should not
be needed, but sometimes my load gets really really high, and when I can
use the system again, there is 2-3 G of swap usage. I haven't found out
yet what this is, it seems to happen when emerging things, maybe related
to having 5 G tmpfs for portage, but when it happened the last time only
100 M were being used.

Yeah so it wont be swap related. This sounds more like the desktop
responsiveness issue discussed a while back. It might be worth
googling that (main issue was fixed in later kernels)
http://www.phoronix.com/scan.php?page=article&item=linux_2637_video&num=1

There's other things that may help, ie
CONFIG_HZ_1000=y
CONFIG_HZ=1000

[Michael Hampicke]

Post by Adam Carter

Post by Alex Schuster
There's plenty of swap space available. With 16 G of RAM it should not
be needed, but sometimes my load gets really really high, and when I can
use the system again, there is 2-3 G of swap usage. I haven't found out
yet what this is, it seems to happen when emerging things, maybe related
to having 5 G tmpfs for portage, but when it happened the last time only
100 M were being used.

Yeah so it wont be swap related. This sounds more like the desktop
responsiveness issue discussed a while back. It might be worth
googling that (main issue was fixed in later kernels)
http://www.phoronix.com/scan.php?page=article&item=linux_2637_video&num=1
There's other things that may help, ie
CONFIG_HZ_1000=y
CONFIG_HZ=1000

Does the CONFIG_HZ setting today really have that much of an impact? I
mean with tickles kernels and high res timers it should matter that
much, or am I wrong?
Playing video is not really a low-latency-multimedia-app, on the other
hand: you never really know :)

[Paul Hartman]

Post by Alex Schuster

Post by Neil Bothwick

Post by Alex Schuster
I guess I could remove anything running on my KDE desktop one by one,
including plasmoids, and see if playback gets better. But not now, I
finally have to actually do some work.

I recently experienced slowdowns and delays with KDE. It turned out I
had inadvertently disabled swap (I'd rearranged my partitions and not
updated fstab). As soon as I gave it some swap space the delays
disappeared.

There's plenty of swap space available. With 16 G of RAM it should not
be needed, but sometimes my load gets really really high, and when I can
use the system again, there is 2-3 G of swap usage. I haven't found out
yet what this is, it seems to happen when emerging things, maybe related
to having 5 G tmpfs for portage, but when it happened the last time only
100 M were being used.

Hi,

I realize this thread is bigger than an encyclopedia by now, so I
apologize if this has already been suggested. :)

I'm curious if you look at /proc/interrupts if the disk with I/O
problems is sharing interrupt with some other device. Maybe there is a
conflict of some sort.

On my motherboard, one of the SATA controllers shares an interrupt
with the soundcard, for example.

[Alex Schuster]

Post by Paul Hartman
I realize this thread is bigger than an encyclopedia by now, so I
apologize if this has already been suggested. :)

Well, I'm happy for any input on this :) This problem is really annoying.

Post by Paul Hartman
I'm curious if you look at /proc/interrupts if the disk with I/O
problems is sharing interrupt with some other device. Maybe there is a
conflict of some sort.
On my motherboard, one of the SATA controllers shares an interrupt
with the soundcard, for example.

Good point. But this does not seem to be the case. The only lines that
have multiple entries are two interrupts with ohci_hdc and snd_hda_intel, that should be no problem. The two ahci entries have their own
interrupts.

I did not yet find the time to remove the plasmoids (all is fine in
other window managers, or for another user with a plain unconfigured KDE
desktop), today I dealt with a horrible and weird qt upgrade experience on
another PC. Finally I was able to downgrade to 4.7, and at least stuff is
working again there.

Wonko

[Alex Schuster]

Finally, I found something. It's Dolphin!

I've done some longer testing, always playing the same video parallel
with a dd if=/dev/zero of=/tmp/10G bs=1M count=10000, in mplayer, for a
minute, several times. When I do this by opening the file in Dolphin, I
get about 15 interruptions, some for longer than a second. Started on the
command line, there are very few, I can play the video for minutes
without a gap. Hooray!

In KDE, I usually play videos by opening them in Dolphin. I exchanged
'mplayer %U' by 'xterm -T MPLAYER -e mplayer %U' in the settings, now
mplayer runs in a terminal, and all is fine. I created a window rule so
the terminal automatically minimizes. Cool!

It only happens in mplayer and mplayer2. Other players work fine, but I
like mplayer best, and prefer to run it without any window decoration.

Now, would this be an MPlayer problem, or one of Dolphin?

Wonko

[Norman Invasion]

Post by Alex Schuster
Finally, I found something. It's Dolphin!
I've done some longer testing, always playing the same video parallel
with a dd if=/dev/zero of=/tmp/10G bs=1M count=10000, in mplayer, for a
minute, several times. When I do this by opening the file in Dolphin, I
get about 15 interruptions, some for longer than a second. Started on the
command line, there are very few, I can play the video for minutes
without a gap. Hooray!
In KDE, I usually play videos by opening them in Dolphin. I exchanged
'mplayer %U' by 'xterm -T MPLAYER -e mplayer %U' in the settings, now
mplayer runs in a terminal, and all is fine. I created a window rule so
the terminal automatically minimizes. Cool!
It only happens in mplayer and mplayer2. Other players work fine, but I
like mplayer best, and prefer to run it without any window decoration.
Now, would this be an MPlayer problem, or one of Dolphin?

Apologies: I haven't followed this thread from the beginning,
but do you have any advanced power management features
enabled (especially hard drive related)?
When I pull the power cord on my lap-top, it goes into all kinds
of nutty "power-saving" and mplayer has long pauses while
the drive spins back up.

[Alex Schuster]

Post by Norman Invasion

Post by Alex Schuster
Finally, I found something. It's Dolphin!

[...]

Post by Norman Invasion
Apologies: I haven't followed this thread from the beginning,

Which was quite long ago :)

Post by Norman Invasion
but do you have any advanced power management features
enabled (especially hard drive related)?

My drives spin down after 30 minutes of idle time, but this never happens
for the system drive. The CPU is set to throttle down from 3600 MHz to
1400 MHz with the ondemand governor, but changing to performance governor
makes no change.

Post by Norman Invasion
When I pull the power cord on my lap-top, it goes into all kinds
of nutty "power-saving" and mplayer has long pauses while
the drive spins back up.

Yeah, but those pauses are much longer than the small interruptions that
are a fraction of a second mostly, and do not happen 15 times per minute.
And it only happens when MPlayer is started from Dolphin. Well, mainly,
when there is much system load, I also had small interruptions when I run
mplayer from the command line, but they are much much less frequent, and
do not happen under normal circumstances, like when doing emerges while
playing videos.

Wonko

[Norman Invasion]

Post by Alex Schuster

Post by Norman Invasion

Post by Alex Schuster
Finally, I found something. It's Dolphin!

[...]

Post by Norman Invasion
Apologies: I haven't followed this thread from the beginning,

Which was quite long ago :)

Post by Norman Invasion
but do you have any advanced power management features
enabled (especially hard drive related)?

My drives spin down after 30 minutes of idle time, but this never happens
for the system drive. The CPU is set to throttle down from 3600 MHz to
1400 MHz with the ondemand governor, but changing to performance governor
makes no change.

Post by Norman Invasion
When I pull the power cord on my lap-top, it goes into all kinds
of nutty "power-saving" and mplayer has long pauses while
the drive spins back up.

Yeah, but those pauses are much longer than the small interruptions that
are a fraction of a second mostly, and do not happen 15 times per minute.
And it only happens when MPlayer is started from Dolphin. Well, mainly,
when there is much system load, I also had small interruptions when I run
mplayer from the command line, but they are much much less frequent, and
do not happen under normal circumstances, like when doing emerges while
playing videos.

I'm just recalling that I get stuttering audio in freebsd, which is caused
by what-I-don't-know, but only happens when the CPU load is low.
Firing up burncpu or doing useless recompiles ameliorates it.

[Alan McKinnon]

This post might be inappropriate. Click to display it.

[Michael Mol]

Post by Alan McKinnon
[1] .avi files are notorious for this shit. It's what happens when you
are Microsoft and you release any old crappy format without consulting
the other experts out there (who will always outnumber you)

Which better container formats were available at the time AVI was
released (1992)? The only contemporary container format I'm aware of
is RIFF, which came out in 1988. MPEG-1 didn't come out until 1993,
which was the same year the Ogg project started. Real's stuff didn't
come out until 1995. Matroska was announced a decade later, in 2005.

Matroska, MP4 and even OGG are nicer container formats, sure, but they
weren't around yet. And even with any of them, it's perfectly possible
to accidentally get A/V desync or stuttering if you don't mux your
streams properly.

(This post draws heavily on Wikipedia for date information, and dates
may be considered only as accurate as Wikipedia...)

[Alan McKinnon]

On Sun, 13 May 2012 14:12:04 -0400

On Sun, May 13, 2012 at 4:56 AM, Alan McKinnon

Post by Alan McKinnon
[1] .avi files are notorious for this shit. It's what happens when
you are Microsoft and you release any old crappy format without
consulting the other experts out there (who will always outnumber
you)

Which better container formats were available at the time AVI was
released (1992)? The only contemporary container format I'm aware of
is RIFF, which came out in 1988. MPEG-1 didn't come out until 1993,
which was the same year the Ogg project started. Real's stuff didn't
come out until 1995. Matroska was announced a decade later, in 2005.
Matroska, MP4 and even OGG are nicer container formats, sure, but they
weren't around yet. And even with any of them, it's perfectly possible
to accidentally get A/V desync or stuttering if you don't mux your
streams properly.
(This post draws heavily on Wikipedia for date information, and dates
may be considered only as accurate as Wikipedia...)

You missed the essence of my post entirely.

[Michael Mol]

Post by Alan McKinnon
On Sun, 13 May 2012 14:12:04 -0400

On Sun, May 13, 2012 at 4:56 AM, Alan McKinnon

Post by Alan McKinnon
[1] .avi files are notorious for this shit. It's what happens when
you are Microsoft and you release any old crappy format without
consulting the other experts out there (who will always outnumber
you)

Which better container formats were available at the time AVI was
released (1992)? The only contemporary container format I'm aware of
is RIFF, which came out in 1988. MPEG-1 didn't come out until 1993,
which was the same year the Ogg project started. Real's stuff didn't
come out until 1995. Matroska was announced a decade later, in 2005.
Matroska, MP4 and even OGG are nicer container formats, sure, but they
weren't around yet. And even with any of them, it's perfectly possible
to accidentally get A/V desync or stuttering if you don't mux your
streams properly.
(This post draws heavily on Wikipedia for date information, and dates
may be considered only as accurate as Wikipedia...)

You missed the essence of my post entirely.

Anti-Microsoft snark? I thought I was calling you on it.

[Alan McKinnon]

On Sun, 13 May 2012 17:01:07 -0400

On Sun, May 13, 2012 at 4:53 PM, Alan McKinnon

Post by Alan McKinnon
On Sun, 13 May 2012 14:12:04 -0400

On Sun, May 13, 2012 at 4:56 AM, Alan McKinnon

Post by Alan McKinnon
[1] .avi files are notorious for this shit. It's what happens
when you are Microsoft and you release any old crappy format
without consulting the other experts out there (who will always
outnumber you)

Which better container formats were available at the time AVI was
released (1992)? The only contemporary container format I'm aware
of is RIFF, which came out in 1988. MPEG-1 didn't come out until
1993, which was the same year the Ogg project started. Real's
stuff didn't come out until 1995. Matroska was announced a decade
later, in 2005.
Matroska, MP4 and even OGG are nicer container formats, sure, but
they weren't around yet. And even with any of them, it's perfectly
possible to accidentally get A/V desync or stuttering if you don't
mux your streams properly.
(This post draws heavily on Wikipedia for date information, and
dates may be considered only as accurate as Wikipedia...)

You missed the essence of my post entirely.

Anti-Microsoft snark? I thought I was calling you on it.

I said .avi is a crappy format, and it is, that much is obvious to
anyone who understands the simple basics of what a container should do.
It would have been obvious to the .avi developers then. And yet it
somehow made it's way to market and got used extensively

You asked what alternatives were available. That is not a question I
asked. It matters nothing that the public used .avi so much (they had
precious little in the way of choice). So whether they had
alternatives or not is irrelevant.

The entire gist of my post was about how .avi as it stands is crappy
and should never have been released by an entity with the engineering
clout of Microsoft as they don't have the excuse of being one dude in
Mom's basement who didn't know better. They really should have known
better.

[Michael Mol]

Post by Alan McKinnon
On Sun, 13 May 2012 17:01:07 -0400

On Sun, May 13, 2012 at 4:53 PM, Alan McKinnon

Post by Alan McKinnon
On Sun, 13 May 2012 14:12:04 -0400

On Sun, May 13, 2012 at 4:56 AM, Alan McKinnon

Post by Alan McKinnon
[1] .avi files are notorious for this shit. It's what happens
when you are Microsoft and you release any old crappy format
without consulting the other experts out there (who will always
outnumber you)

Which better container formats were available at the time AVI was
released (1992)? The only contemporary container format I'm aware
of is RIFF, which came out in 1988. MPEG-1 didn't come out until
1993, which was the same year the Ogg project started. Real's
stuff didn't come out until 1995. Matroska was announced a decade
later, in 2005.
Matroska, MP4 and even OGG are nicer container formats, sure, but
they weren't around yet. And even with any of them, it's perfectly
possible to accidentally get A/V desync or stuttering if you don't
mux your streams properly.
(This post draws heavily on Wikipedia for date information, and
dates may be considered only as accurate as Wikipedia...)

You missed the essence of my post entirely.

Anti-Microsoft snark? I thought I was calling you on it.

I said .avi is a crappy format, and it is, that much is obvious to
anyone who understands the simple basics of what a container should do.

The MPEG group had only been formed four years prior to AVI's release,
and didn't release their first standard until a year later. Meanwhile,
Microsoft needed a video file format that:

1) Was a file format that sat on disk
2) Synchronized audio and video
3) Integrated cleanly with their being-developed operating system (AVI
is very closely related to the Video for Windows API. It's worth
noting that WMF, another Microsoft format from this time, is
essentially a serialized form of their drawing primitives.)
4) Ran smoothly on an 80386 at 33MHz with a 16-bit, 8MHz data bus
between the CPU and persistent storage.

With the exception of perhaps (3), those are the "basics." Consider
that this was released in 1992, and then consider that it had probably
been under development for at least a couple years prior.

I won't disagree that AVI is a crappy format by today's standards, and
that it should be avoided where possible, but what you consider simple
and obvious today was *new* at the time, and so not simple and
obvious.

Post by Alan McKinnon
It would have been obvious to the .avi developers then. And yet it
somehow made it's way to market and got used extensively
You asked what alternatives were available. That is not a question I
asked. It matters nothing that the public used .avi so much (they had
precious little in the way of choice). So whether they had
alternatives or not is irrelevant.

It's entirely relevant if you want to consider whether not the
expertise to come up with a 2012-modern format *existed* in the
lead-up time to 1992.

Post by Alan McKinnon
The entire gist of my post was about how .avi as it stands is crappy
and should never have been released by an entity with the engineering
clout of Microsoft as they don't have the excuse of being one dude in
Mom's basement who didn't know better. They really should have known
better.

Seriously, why? Why do you think that the entire engineering clout of
a company which hadn't yet taken over the desktop market(!) would be
focused on perfecting AVI, one piece of a large,
already-late-to-market product? They had a bunch of difficult things
to pay attention to, such as mixing protected-mode and real-mode
applications on hardware in a task-switching environment, and working
around compatibility for programs whose developers still assumed they
had full run of the system. On a 386.

[Alan McKinnon]

On Sun, 13 May 2012 18:03:59 -0400

On Sun, May 13, 2012 at 5:33 PM, Alan McKinnon

Post by Alan McKinnon
On Sun, 13 May 2012 17:01:07 -0400

On Sun, May 13, 2012 at 4:53 PM, Alan McKinnon

Post by Alan McKinnon
On Sun, 13 May 2012 14:12:04 -0400

On Sun, May 13, 2012 at 4:56 AM, Alan McKinnon

Post by Alan McKinnon
[1] .avi files are notorious for this shit. It's what happens
when you are Microsoft and you release any old crappy format
without consulting the other experts out there (who will
always outnumber you)

Which better container formats were available at the time AVI
was released (1992)? The only contemporary container format I'm
aware of is RIFF, which came out in 1988. MPEG-1 didn't come
out until 1993, which was the same year the Ogg project
started. Real's stuff didn't come out until 1995. Matroska was
announced a decade later, in 2005.
Matroska, MP4 and even OGG are nicer container formats, sure,
but they weren't around yet. And even with any of them, it's
perfectly possible to accidentally get A/V desync or stuttering
if you don't mux your streams properly.
(This post draws heavily on Wikipedia for date information, and
dates may be considered only as accurate as Wikipedia...)

You missed the essence of my post entirely.

Anti-Microsoft snark? I thought I was calling you on it.

I said .avi is a crappy format, and it is, that much is obvious to
anyone who understands the simple basics of what a container should do.

The MPEG group had only been formed four years prior to AVI's release,
and didn't release their first standard until a year later. Meanwhile,
1) Was a file format that sat on disk
2) Synchronized audio and video

This is the part they got wrong.

Would you not agree that this is the second-most important feature
required, where the ability to actually play the audio/video at all is
the first?

Getting that wrong is to me akin to building a car and forgetting to
provide it with an adequate means of stopping. There are many other
things that can be forgiven where one would need a predictive crystal
ball, but needing time sync information in the container is just simply
self-evident.

3) Integrated cleanly with their being-developed operating system (AVI
is very closely related to the Video for Windows API. It's worth
noting that WMF, another Microsoft format from this time, is
essentially a serialized form of their drawing primitives.)
4) Ran smoothly on an 80386 at 33MHz with a 16-bit, 8MHz data bus
between the CPU and persistent storage.
With the exception of perhaps (3), those are the "basics." Consider
that this was released in 1992, and then consider that it had probably
been under development for at least a couple years prior.
I won't disagree that AVI is a crappy format by today's standards, and
that it should be avoided where possible, but what you consider simple
and obvious today was *new* at the time, and so not simple and
obvious.

I'm not talking about today's standards. I'm talking about 1992
standards.

It's not reasonable to expect MS devs to anticipate algorithms that did
not exist then, or hardware that was 10 years away, or even that the
internet would be what it is. I do expect devs to get right aspects of
their software that will be used right at the time it is released.

Post by Alan McKinnon
It would have been obvious to the .avi developers then. And yet it
somehow made it's way to market and got used extensively
You asked what alternatives were available. That is not a question I
asked. It matters nothing that the public used .avi so much (they
had precious little in the way of choice). So whether they had
alternatives or not is irrelevant.

It's entirely relevant if you want to consider whether not the
expertise to come up with a 2012-modern format *existed* in the
lead-up time to 1992.

Again, I'm not talking about 2012

Post by Alan McKinnon
The entire gist of my post was about how .avi as it stands is crappy
and should never have been released by an entity with the
engineering clout of Microsoft as they don't have the excuse of
being one dude in Mom's basement who didn't know better. They
really should have known better.

Seriously, why? Why do you think that the entire engineering clout of
a company which hadn't yet taken over the desktop market(!) would be
focused on perfecting AVI, one piece of a large,
already-late-to-market product? They had a bunch of difficult things
to pay attention to, such as mixing protected-mode and real-mode
applications on hardware in a task-switching environment, and working
around compatibility for programs whose developers still assumed they
had full run of the system. On a 386.

No, I expect them to get the basics right. Cars and brakes.

[Michael Mol]

Post by Alan McKinnon
On Sun, 13 May 2012 18:03:59 -0400

On Sun, May 13, 2012 at 5:33 PM, Alan McKinnon

Post by Alan McKinnon
On Sun, 13 May 2012 17:01:07 -0400

On Sun, May 13, 2012 at 4:53 PM, Alan McKinnon

Post by Alan McKinnon
On Sun, 13 May 2012 14:12:04 -0400

On Sun, May 13, 2012 at 4:56 AM, Alan McKinnon

Post by Alan McKinnon
[1] .avi files are notorious for this shit. It's what happens
when you are Microsoft and you release any old crappy format
without consulting the other experts out there (who will
always outnumber you)

Which better container formats were available at the time AVI
was released (1992)? The only contemporary container format I'm
aware of is RIFF, which came out in 1988. MPEG-1 didn't come
out until 1993, which was the same year the Ogg project
started. Real's stuff didn't come out until 1995. Matroska was
announced a decade later, in 2005.
Matroska, MP4 and even OGG are nicer container formats, sure,
but they weren't around yet. And even with any of them, it's
perfectly possible to accidentally get A/V desync or stuttering
if you don't mux your streams properly.
(This post draws heavily on Wikipedia for date information, and
dates may be considered only as accurate as Wikipedia...)

You missed the essence of my post entirely.

Anti-Microsoft snark? I thought I was calling you on it.

I said .avi is a crappy format, and it is, that much is obvious to
anyone who understands the simple basics of what a container should do.

The MPEG group had only been formed four years prior to AVI's release,
and didn't release their first standard until a year later. Meanwhile,
1) Was a file format that sat on disk
2) Synchronized audio and video

This is the part they got wrong.
Would you not agree that this is the second-most important feature
required, where the ability to actually play the audio/video at all is
the first?

You're going to have to go into detail. Last I checked, old versions
of Windows shipped with AVI files for their animations, and those AVI
files played fine. So it _sounds_ like they're able to play video, at
least.

And my largish collection of AMVs and videos I've put together myself
suggest that AVI can play synchronized audio and video.

Post by Alan McKinnon
Getting that wrong is to me akin to building a car and forgetting to
provide it with an adequate means of stopping. There are many other
things that can be forgiven where one would need a predictive crystal
ball, but needing time sync information in the container is just simply
self-evident.

Only if you anticipate your audio and video streams deviating from
intended usages. AVI is used for far more things than it was designed
to do. Reading deeper into its history, it sounds like it was embraced
and extended by entities outside of Microsoft to do things it wasn't
designed for in the first place. So expecting it to handle VBR audio
or video with predictive frames is kinda like putting a supercharger
in a Pinto and complaining when it winds up sitting on its own roof.

Post by Alan McKinnon

3) Integrated cleanly with their being-developed operating system (AVI
is very closely related to the Video for Windows API. It's worth
noting that WMF, another Microsoft format from this time, is
essentially a serialized form of their drawing primitives.)
4) Ran smoothly on an 80386 at 33MHz with a 16-bit, 8MHz data bus
between the CPU and persistent storage.
With the exception of perhaps (3), those are the "basics." Consider
that this was released in 1992, and then consider that it had probably
been under development for at least a couple years prior.
I won't disagree that AVI is a crappy format by today's standards, and
that it should be avoided where possible, but what you consider simple
and obvious today was *new* at the time, and so not simple and
obvious.

I'm not talking about today's standards. I'm talking about 1992
standards.

_Those standards didn't exist._ That's been my key point.

Yes, there was SMPTE, but that's for video recording and production
houses, and that was certainly not a planned usage for AVI.

Post by Alan McKinnon
It's not reasonable to expect MS devs to anticipate algorithms that did
not exist then, or hardware that was 10 years away, or even that the
internet would be what it is. I do expect devs to get right aspects of
their software that will be used right at the time it is released.

The earliest AVI files I'm aware of were sequences of RLE bitmaps, and
the code doing playback knew *exactly* what the framerate was, because
it knew what the video was for. Framerate support was added by
external parties because external parties wanted to extend AVI for
their own purposes. For that matter, AVI was an extension of RIFF.

Post by Alan McKinnon

Post by Alan McKinnon
It would have been obvious to the .avi developers then. And yet it
somehow made it's way to market and got used extensively
You asked what alternatives were available. That is not a question I
asked. It matters nothing that the public used .avi so much (they
had precious little in the way of choice). So whether they had
alternatives or not is irrelevant.

It's entirely relevant if you want to consider whether not the
expertise to come up with a 2012-modern format *existed* in the
lead-up time to 1992.

Again, I'm not talking about 2012

No, but you're talking from the perspective of 2012, with a 20-year hindsight.

Post by Alan McKinnon

Post by Alan McKinnon
The entire gist of my post was about how .avi as it stands is crappy
and should never have been released by an entity with the
engineering clout of Microsoft as they don't have the excuse of
being one dude in Mom's basement who didn't know better. They
really should have known better.

Seriously, why? Why do you think that the entire engineering clout of
a company which hadn't yet taken over the desktop market(!) would be
focused on perfecting AVI, one piece of a large,
already-late-to-market product? They had a bunch of difficult things
to pay attention to, such as mixing protected-mode and real-mode
applications on hardware in a task-switching environment, and working
around compatibility for programs whose developers still assumed they
had full run of the system. On a 386.

No, I expect them to get the basics right. Cars and brakes.

Pintos and superchargers. AVI does far more than what it was designed
for, and even what you want it to have been designed to do wasn't even
in the forecast in 1992. In 1992, the biggest and most interesting
applications were still conceptual derivatives of VisiCalc. (Actually,
that's probably still true today, but in 1992 there wasn't as big a
base of software developers to work other concepts.)

[Paul Hartman]

Post by Alex Schuster
Finally, I found something. It's Dolphin!
I've done some longer testing, always playing the same video parallel
with a dd if=/dev/zero of=/tmp/10G bs=1M count=10000, in mplayer, for a
minute, several times. When I do this by opening the file in Dolphin, I
get about 15 interruptions, some for longer than a second. Started on the
command line, there are very few, I can play the video for minutes
without a gap. Hooray!
In KDE, I usually play videos by opening them in Dolphin. I exchanged
'mplayer %U' by 'xterm -T MPLAYER -e mplayer %U' in the settings, now
mplayer runs in a terminal, and all is fine. I created a window rule so
the terminal automatically minimizes. Cool!
It only happens in mplayer and mplayer2. Other players work fine, but I
like mplayer best, and prefer to run it without any window decoration.
Now, would this be an MPlayer problem, or one of Dolphin?

I wonder if Dolphin is generating thumbnails/preview indexes at the
same time you're trying to play, causing resource contention. I wonder
if you can disable thumbnail generation by Dolphin or remove the
association it has with video files and mplayer.

[Alex Schuster]

Post by Paul Hartman

Post by Alex Schuster
Finally, I found something. It's Dolphin!

[...]

Post by Paul Hartman

Post by Alex Schuster
Now, would this be an MPlayer problem, or one of Dolphin?

I wonder if Dolphin is generating thumbnails/preview indexes at the
same time you're trying to play, causing resource contention.

Sometimes it does that, but that would happen when I play from the command
line, too. And now it even works from Dolphin, when it is set to open
mplayer in a terminal. So this cannot be the problem. And even if it
were, avoiding thumbnails would be a bad workaround only, such operations
should not affect video playback.

Maybe this is somehow related to the other problem I had with mplayer2
only, using 100% CPU when idle, only when started from a file manager.

Wonko

[Dale]

Alex Schuster wrote:
<<< SNIP >>>

Post by Alex Schuster
Now this is really annoying. I watch small clips mostly, and can live
with that, and when I want to watch stuff with others, I copy the file to
tmpfs, which seems to help a lot.
But now I found another solution: NOT USING KDE.
When X crashed (trying to make the old Unreal game play), I fired up
another window manager, and when I played a video in there, there was no
problem at all. So, I have another workaround.
But does anyone have an idea, why running KDE is the problem? Disabling
desktop effects does not help.
I must be totally crazy because I still want to use KDE, despite the big
trouble it gives me nearly every day. Yes, most things work fine now, but
there are many many little problems, daily application crashes, and every
time I log in I fear that the desktop won't come up. 8G of RAM was not
enough to avoid swapping, so now I have 16G, that's fine, I no longer
care about kwin using 1G of my RAM. Oh, and I no longer use KMail, after
it ate thousands of mails I just wanted to move. No problem, they were
not important, but I no longer trust the KDEPIM suite. And it seems the
developers do not care about this, the bug report got no replies.
But anyway. Any idea why it only happens with KDE? I will ask on the KDE
mailing list, but I thought I post here first, maybe there's something
Gentoo-specific going on here.
Wonko

When I first built this rig, I ran into this issue as well. What I did
was tell smplayer, in my case, to cache more of the video. I have mine
set to cache 6Mbs and it plays fine even on HD videos.

This may not help you but if you have not tried it yet, may be worth a
shot. It is aggravating when it does this tho.

Also, there was a thread a good while back with this issue and their fix
was to do a emerge -e world with everything optimized for their CPU and
such. May be worth thinking about at least.

Dale

:-) :-)

[Alex Schuster]

Post by Dale
When I first built this rig, I ran into this issue as well. What I did
was tell smplayer, in my case, to cache more of the video. I have mine
set to cache 6Mbs and it plays fine even on HD videos.

I have cache = 131072 and cache-min=20.0 in .mplayer/config. That's
128MB, this should be enough. Got this hint by James Broadhead some
months ago in this thread.

Post by Dale
Also, there was a thread a good while back with this issue and their fix
was to do a emerge -e world with everything optimized for their CPU and
such. May be worth thinking about at least.

That was Walter Dnes, also in thos thread. But everything should be quite
optimized here, with:
CFLAGS="-pipe -march=amdfam10 -O2 \
-floop-interchange -floop-strip-mine -floop-block \
-msse -msse2 -msse3 -msse4a -msse4.1 -msse4.2 -m3dnow \
-mcx16 -msahf -maes -mpclmul -mpopcnt -mabm -mlwp -mavx \
--param l1-cache-size=16 \
--param l1-cache-line-size=64 \
--param l2-cache-size=2048"

And I do not have high CPU load when playing movies.

Wonko

[Stroller]

Post by Dale
...
Also, there was a thread a good while back with this issue and their fix
was to do a emerge -e world with everything optimized for their CPU and
such. May be worth thinking about at least.

I understood that issue as significantly different - in that case there was stuttering of demanding video with no other i/o occurring. In this case *any* video, even low-res standard-def video, is pausing when i/o load is applied. The previous case is CPU / GPU video decoding performance issue, I think this one is clearly not.

Stroller.

[Simon]

Post by Dale
Also, there was a thread a good while back with this issue and their fix
was to do a emerge -e world with everything optimized for their CPU and
such.  May be worth thinking about at least.

Video playback and CPU optimisations go hand in hand.

One video I have had for a long time, I could never play it at all on
any of my computers and I could not beleive what I saw next...
I got a new PC and thought to give it a try. It worked painfully,
skipping and using 100% of one of my CPU cores. I was sad and thought
this video might have been corrupt. After a wave of cleanup, I had
removed all CPU optimisation flags from my make.conf (I was originally
negating many, like -sse, etc, so I left it "to my profile" to choose
what was needed) and recompiled everything. After that, the same
movie played in the best quality I had ever seen and CPU usage was
below 5% !!!! I was even able to open 12 movies (4x3) all playing at
the same time (this was unthinkable before I optimized my system).
Anyway, keep in mind that my particular starting scenario was that I
had "un-optimized" it long ago!

Good luck!