ffmpeg not creating previews in Newznab

Had an issue on my system where ffmpeg latest version installed via brew (2.2.1) yet I couldn't get any previews on media.

Much head scratching later, it turns out that because my ffmpeg install is so new newznab wasn't configured to use it.

So, you need to edit postprocess.php & add the following line (//2.x):

Make sure you comment out the other lines as above, its near the bottom of the file! :-)

Running Newznab on OSX Mac Mini Mavericks Server using XAMPP

After using an UBUNTU VM I switched to running newznab on a pure OSX install (Mavericks Server), due to the issues I had with the VM crashing on occasion...plus I thought the extra resources it consumed made sense to have it running direct.

Installation was very straightforward, using the below:

1. I decided to use an XAMPP installation while testing this setup but am soon switching over to using the Mavericks Server setup (simply as it one less point of failure...and why not!)

2. Get the newznab install - plus edition is a no-brainer as regular regexes mean lots of content :-)
http://www.newznab.com/

3. Install XAMPP http://www.apachefriends.org/en/xampp-macosx.html, its a simple drag to Applications

4. Install newznab, svn install with site details from newznab...the usual! Place the install into the www folder of XAMPP

5. In XAMPP control panel start:

MySQL
Apache web server

Once they have a green light next to them you can open Safari web browser and navigate to:

localhost or 127.0.0.1

And you should see the welcome screen:

Congrats, so far so good! now shutdown both the MySQL & Apache by clicking stop & wait till they turn red.

6. Configure XAMPP:

Open Applications/XAMPP/etc/httpd.conf

Best to edit using some like Textwranger

You need to edit around line 180, this is to tell your web server to send any web requests to newznab by default....you can use Apache to host multiple sites quite easily but I will discuss this in another post.

DocumentRoot "/Applications/XAMPP/xamppfiles/htdocs/nnplus/www/"

Make sure your changes are exactly as above!

Close & save the file, then restart MySQL & Apache in XAMPP control panel.

Open Safari and type localhost into address bar - you should now see newznab but with an error...don't worry you're doing good!

7. Configure PHP

Open Applications/XAMPP/etc/php.ini

The max_execution_time needs to be increased – suggest 120
The memory_limit needs to be increased – suggest 1024M

Save the file.

8. Complete Newznab install, set correct folder permissions:

Open a terminal session and copy/paste each line below, pressing enter after each paste to set permissions

chmod 777 /Applications/XAMPP/xamppfiles/htdocs/nnplus/www/lib/smarty/templates_c

chmod 777 /Applications/XAMPP/xamppfiles/htdocs/nnplus/www/covers/movies

chmod 777 /Applications/XAMPP/xamppfiles/htdocs/nnplus/www/covers/anime

chmod 777 /Applications/XAMPP/xamppfiles/htdocs/nnplus/www/covers/music

chmod 777 /Applications/XAMPP/xamppfiles/htdocs/nnplus/www

chmod 777 /Applications/XAMPP/xamppfiles/htdocs/nnplus/www/install

chmod -R 777 /Applications/XAMPP/xamppfiles/htdocs/nnplus/nzbfiles/

In the XAMPP control panel, ensure MySQL & Apache lights are green.

Open Safari & goto localhost (127.0.0.1) which should now show the newznab install screen with no errors, click the button "Go to step: Pre Flight Check"

Long  as you followed steps 6 & 7 you will then be error free and click "Go to step two: Set up the database"

You will see

Default user is root and password blank unless you have changed it in XAMPP

Click "setup database"

It will setup the database and present you with a screen telling you the database setup is correct

Click the button showing “Step three: Setup News Server connection”

You should have a newsgroup provider setup and have the login/connection details for it – type them in here and click test connection.

You should receive a message telling you “The news server setup is correct…..”

Click the button stating ”Step four: Cache Settings”

Click “Save configuration file”

Click the button marked “Step 5: setup admin user”

Fill in this section and then click the button.

Then click the button marked “Step Seven: Set NZB File Path”

Use the default path and so long as you have correctly entered all the chmod commands presented earlier in step 8 you should get no errors – if you get an error follow the instructions in the error.

Otherwise click the button marked “Set NZB File Path”

Install complete, now click on the admin home page link to begin the admin setup.

9. Admin Setup Newznab

Click on the site options link.

Paste your newznab ID from your welcome email into the last field of the first section.

If your newsgroup provide supports compressed headers (not many do) in the section marked Usenet Settings select the Use compressed headers toggle – do not select if not supported!

Change Max Messages to 60,000 I find this runs pretty smooth but you can experiment with higher or lower figures (I did try @ 100,000 but my system kept barfing...could need a MySQL tweak on memory settings as my mac has 16GB sweeeet :-))

In the Where to start new groups field I change the toggle box to days and enter 1 day (initially it will go back one day to get all the headers – that is enough to start off your site) – if you leave the 20000 posts in there then you will not have enough posts to complete an episode or MP3.

Then scroll to the bottom and click “Save site settings”

10. Select Newsgroups to index

I personally use these, they seem to grab enough releases and things move fast enough..pick or leave em at your leisure!

a.b.x264

a.b.tvseries

a.b.tv

a.b.town

a.b.teevee

a.b.multimedia

a.b.movies.xvid

a.b.movies.divx

a.b.movies

a.b.moovee

a.b.mom

a.b.hdtv.x264

a.b.hdtv

a.b.ghosts

a.b.ebook

a.b.e-book.technical

a.b.e-book.magazines

a.b.e-book.flood

a.b.e-book

a.b.comics.dcp

a.b.astronomy

a.b.0day.stuffz

11. Get things running!

So there are two files that need to be run, 1 will grab releases & the other will preform all the updates that turn the files into the nice content we are looking for!

Open a terminal session

cd /Applications/xampp/xamppfiles/htdocs/nnplus/misc/update_scripts

/applications/xampp/xamppfiles/bin/php-5.3.1 update_binaries.php

The above looks for headers and starts to process them based on regexes, once this has completed you need to run the following

cd /Applications/xampp/xamppfiles/htdocs/nnplus/misc/update_scripts

/applications/xampp/xamppfiles/bin/php-5.3.1 update_releases.php

This turns the millions of headers into releases and populates all the sections in your Newznab (Movies, TV, Books etc....)

12. Making Life Easy!

The best way to run the above though is through an automated script such as the following:

In Applications/XAMPP/xampp/htdocs/nnplus/misc/update_scripts/nix_scripts

You should have a file newznab_local.sh - copy this file copy newznab.local.sh newznab_update.sh

Then edit the newznab_update.sh file in Textwranger to look like:

#!/bin/sh

# call this script from within screen to get binaries, processes releases and 

# every half day get tv/theatre info and optimise the database

set -e

export NEWZNAB_PATH="/Applications/XAMPP/xamppfiles/htdocs/nnplus/misc/update_scripts"

export NEWZNAB_SLEEP_TIME="600" # in seconds

export PHP_PATH="/Applications/XAMPP/xamppfiles/bin"

LASTOPTIMIZE=`date +%s`

while :

 do

CURRTIME=`date +%s`

cd ${NEWZNAB_PATH}

${PHP_PATH}/php-5.5.3 ${NEWZNAB_PATH}/update_binaries.php

${PHP_PATH}/php-5.5.3 ${NEWZNAB_PATH}/update_releases.php

DIFF=$(($CURRTIME-$LASTOPTIMIZE))

if [ "$DIFF" -gt 43200 ] || [ "$DIFF" -lt 1 ]

then

   LASTOPTIMIZE=`date +%s`

   ${PHP_PATH}/php-5.5.3 ${NEWZNAB_PATH}/optimise_db.php

   ${PHP_PATH}/php-5.5.3 ${NEWZNAB_PATH}/update_tvschedule.php

   ${PHP_PATH}/php-5.5.3 ${NEWZNAB_PATH}/update_theaters.php

fi

echo "waiting ${NEWZNAB_SLEEP_TIME} seconds..."

sleep ${NEWZNAB_SLEEP_TIME}

done

Save the file

Run the file ./newznab_local.sh

This will then run quite happily and process releases, you do need to either leave the terminal window open so it runs or use something like screen to have it running as a process - something I will discuss in another post.

Video Modes running Ubuntu 12.10 server in Parallels & screen blanking issue

Had some trouble in changing the terminal resolution even after installing the Parallels Tools, while the tools install allowed me to see my shared folders it didn't offer any way of changing the (tiny) 800*600 default screen...sooooo:

First determine the resolution you wish to have the console run at. Most cards support the standard VESA resolutions and their associated mode numbers. Pick the resolution you want and remember it's mode number. I wanted 1024x768x16 so my mode number is 791.

Type sudo su at the terminal prompt and enter your password. This leaves you with a root capable shell. 

Edit /etc/default/grub with:

nano /etc/default/grub

Find the line with GRUB_CMDLINE_LINUX_DEFAULT and add vga=xxx where xxx is the mode number. It should look something like:

GRUB_CMDLINE_LINUX_DEFAUT="quiet vga=791"

Save the changes and exit the editor

Update grub with the new settings. The settings will not take affect with out this.

update-grub

Restart your machine with the reboot command if you are still elevated or su reboot 

*****I choose 794 personally (1,280*1,024, 16bit) but goto Vesa Modes and fill your boots :-)

Is your screen blanking out after about 5-10 mins and hiding all those nice tmux screens!!!??

Edit /etc/default/grub with:

nano /etc/default/grub

Find the line with GRUB_CMDLINE_LINUX_DEFAULT and add consoleblank=0. It should look something like:

GRUB_CMDLINE_LINUX_DEFAUT="quiet consoleblank=0 vga=791"

then run "update-grub" & reboot....no more blank screen (epic win!) :-)

Newznab, Tmux, Backfill & Running on boot (Phew!)

Too many things to type in everytime you reboot your server? Want to backfill a buttload of nzb's you've torrented or been given? Follow the guide below, it automates everything into 1 script & then using tmux shows what's happening:

Step 1:


Postprocess script.

This script only does postprocessing. By default update_releases.php post processes the last 100 releases. if you backfill or if your update_releases.php has more than 100 releases some may not process. This script is to be run in another shell and only does post processing on releases that have not been done yet. when there is nothing to to it looks like it is hung but if you have more releases the script continues. Another rational is that it allows update_releases to finish quicker because the number of post processing it does is changed to 1.


nano +333 /var/www/newznab/www/lib/postprocess.php << change the 100 to a 1.

link to script download (place in update_scripts) justpostprocess

chmod 755 justpostprocessing.php << you can now run this script in a another shell.


Step 2:

The requirements to run this are as follows:

you must have followed the instructions for the Post-Processing script (above)
you must have installed sysstat (apt-get install sysstat)
you must have instaled ifstat (apt-get install ifstat)
you must have a basic understanding of how Tmux works (really really basic if you follow this guide :-)

Step 3:

If you go to your newznab installation folder, (mine for future reference is /var/www/newznab) and look in the misc/testing folder. You’ll see a script called nzb-importmodified.php. Copy that to /where/your/newznab/is/www/admin:

cp /where/your/newznab/is/misc/testing/nzb-importmodified.php /where/your/newznab/is/www/admin

Go to your screen script (or batch script on windows, then nano (or your choice of editor):

nano /where/your/newznab/is/misc/update_scripts/nix_scripts/newznab_screen.sh

Inside there, you should add a reference to your new import modified, with a link to where your nzbs are stored. Here is what my newznab_screen.sh looks like:

newznab_screen

By adding true to the end of the request, it uses the name from the NZB file and doesn’t re-look it up.

create a folder in /var/www/newznab - mkdir tempnzbs 

(This is where you would store any nzb's you want to do a backfill import on....)

Step 4:

Copy this script into /var/www/newznab

NN_Autostart

you must modify line 20 with the appropriate newznab_screen_backfill.sh script for your configuration

(in step 3 I created a copy of newznab_screen.sh & renamed it to newznab_screen_backfill.sh)

Step 5:

run your new startup script:

./NN_Autostart

and watch the magic happen!

If you want to end the session (maybe to run an update etc..) ctrl+c to exit a screen then "tmux kill-server"

PS. If not using Sphinx (why not!) then just edit it out of the script above....same with any information windows you do not require)

Newznab PHP Memory Errors

Since the last big update have started to have Newznab crash out with the old "Allowed memory size exhausted" error so have edited the following scripts:

sudo nano /etc/php/cli/php.ini

Look for memory_limit, and either set it to 256MB, or -1 for no limit:

memory_limit = -1

(I had it set to 256M but now trying -1...systems has 1GB in total ram on the virtual)

Also

apache php.ini
sudo nano /etc/php5/apache2/php.ini

and find memory_limit and set it so
memory_limit = -1

Newznab Updates ***New***

**Update Vr1.1***

Please note that if you do not run number 5 below then your site will break...leaving lots of nice blank pages to look at...which depending on if you enjoy watching snowstorms could be quite fun!

Well they have been coming thick and fast!

Here's a quick breakdown of what I did and all ran pretty smooth:

SVN Update currently showing alt revision of 2707

1.   svn update (in /var/www/newznab/)

2.   Deleted cached files:
...nnplus\www\lib\smarty\templates_c\

(ubuntu so rm *.* in above folder)

3.   DIDN'T import the new regex.sql or schema.sql
(this would overwrite your database)

4 Checked my "php.ini" file for this line and made sure it was active
extension=php_mysqli.dll

5 Ran the database conversion script:
...nnplus\misc\update_scripts
php update_database_version.php 

Results:

...executing patch 2012_12_29_anidb.sql
...executing patch 2013_01_04_host.sql
...executing patch 2013_01_05_category.sql
...executing patch 2013_01_05_unused_parts_index.sql
...executing patch 2013_01_14_nzb2nfo.sql
...executing patch 2013_01_17_templates.sql
...executing patch 2013_01_18_users.sql
...executing patch 2013_01_19_partsbinco1.sql

NewzNab Plus Blacklists

Version 2

Here's how mine is setup, comments would be nice. Here's a link for the text file & sql import versions:

Blacklist Download

Since implementing this, I've had no memory issues especially regarding the hdnectar HUGE blueray rips that were constantly crashing my box!